Content uploaded by Azra Naseem
Author content
All content in this area was uploaded by Azra Naseem on Jun 08, 2019
Content may be subject to copyright.
Abstract—This paper describes the process of developing a
quality assurance checklist for course designers and faculty
members who are engaged in designing blended learning
courses at a multi-campus private university. The purpose is to
establish unified quality criteria for ensuring quality of the
design of blended learning course content. Six categories were
identified in the checklist to meet the purpose: 1) contact
information, copyrights, 2) privacy and security of information,
3) nature of interactive online content, 4) usability, 5)
multimedia, and 6) interface design and technical specifications.
Four stages of the checklist development were followed during
the process: initial discussion for item adaption, validity and
reliability assessment, tool/checklist refinement and finalization.
These stages were found to be useful in creating the quality
checklist for designing the course content taught through
blended learning approaches.
Index Terms—Blended learning, course design, checklist
development, quality assurance.
I. INTRODUCTION
The Aga Khan University (AKU) is an international
University spread across 8 geographical sites in three
continents (Africa, Asia and Europe). To enable increased
collaborative teaching across campuses, AKU initiated a
blended learning programme. In a blended learning approach,
a combination of synchronous and asynchronous technologies
is used to offer courses.
In order to enable faculty members to design and teach
courses through a BL approach, a professional development
programme is offered. The programme itself is designed in
the blended learning format in order to give practical
experience of this approach to the faculty members. The
programme consists of a four-week online learning phase and
a face-to-face workshop of two weeks, where faculty
members participate in various activities to gain knowledge
and skills of designing and teaching through blended learning
approaches. After the face-to-face workshop, the
faculty-participants are facilitated and mentored by a team of
eLearning designers and developers, to redesign an existing
face-to-face course and offer it in a blended format.
There are a number of faculty members and course
designers working on course design and development within
the university. In order to ensure consistency of design
features across the courses, a need was identified to establish
unified quality criteria that can be followed by all
Manuscript received August 30, 2014; revised October 22, 2014.
The authors are with Aga Khan University, Pakistan (e-mail:
sahreen.chauhan@aku.edu, azra.naseem@aku.edu,
eman.rashwan@aku.edu).
stakeholders. In this paper the process of developing the
quality assurance checklist will be described.
II. DEFINING QUALITY E-LEARNING / BLENDED LEARNING
COURSES
In order to ensure successful implementation and
continuous improvement of teaching through BL approaches
it is important for the institutions to undertake various
research and quality assurance initiatives. Thus, to understand
what is meant by „quality‟ and how it is defined for the courses
offered at the institution becomes the most significant step [1].
Defining a quality assurance process is challenging also
because it requires the institution to focus not only on learning
outcomes, but also on the process of improving course design
and teaching using ICT [2].
Many Higher Education institutions have developed
quality review processes for blended learning courses and
implement them on a routine basis[3], [4]. For example,
Centre for Teaching and Educational Technologies (CTET) at
Royal Road University conducted a pilot project to create a
review process for online courses, which are now part of their
regular practice to ensure ongoing improvement. Six
independent but interrelated components were identified for
the online quality courses by CTET: 1) curriculum design, 2)
teaching and facilitation, 3) learning facilitation, 4)
instructional design, 5) web design and 6) course presentation
[3].
Ratnasingam (2014) in her study examined quality
indicators related to the online course design and delivery and
its impact on students‟ learning [4]. These components are
course structure, course content, course navigation and course
assessments.
Quality and standards are used differently in different
educational contexts. Quality in teaching and course design in
eLearning may refer to course content with clear objectives;
teaching activities, strategies and resources linked to the
course objectives; contact information and the ways to
correspond with the teacher; information on the copyright and
information security [5].
In 1987, Gamson and Chickering proposed seven
research-based principles of good teaching practice in
Colleges and Universities. These are: 1) encourages contact
between students and faculty, 2) develops reciprocity and
cooperation among students, 3) encourages active learning, 4)
gives prompt feedback, 5) emphasizes time on task, 6)
communicates high expectations, 7) respects diverse talents
and ways of learning. These principles are applicable to
online and blended learning environments as well [6]. Dayton
and Vaughn (2007) established a quality assurance process
Developing a Quality Checklist for Designing Blended
Learning Course Content
Sahreen Chauhan, Azra Naseem, and Eman Rashwan
International
Journal of Information and Education Technology, Vol. 6, No. 3, March 2016
224
DOI: 10.7763/IJIET.2016.V6.689
using these seven principles as a framework and developed a
checklist to review their online courses [7]. The checklist was
meant to be used by the faculty members as good practice
guidelines in designing new or revising their existing online
courses and also by the instructional designers to carry out the
QA review of the courses. The authors found that the QA
framework needs to be comprehensive, concise and adaptable
to all type of course (including course which are not taught
online) to promote good practices. Moreover, a framework
(based on seven principles) , as in this case, was found to be
useful for developing course and faculty evaluations
questionnaires.
Whilst most of the checklists were about evaluating the
quality of the entire process of course design, a need was felt
to develop a checklist that could guide the process of
designing course content.
III. PROCESS
A checklist was developed to ascertain the quality of the
course content designed by the course designers and the
faculty members. The process of developing the tool
comprised four steps:
1) Initial decisions for item adaptation
2) Validity assessment
3) Reliability assessment
4) Tool refinement and finalization
A. Initial Decisions for Item Adaptation
The process began with a discussion amongst course
designers and faculty members on what constitutes quality of
course content in a blended format. Subsequently, a list of
items was developed by the two course designers, and three
tools were considered more closely.
The first one was shared by designer A (third author of this
paper) that she had used in her previous content development
work for evaluating the eLearning content and online courses
of different higher education institutions and corporates in the
Middle East. The checklist had 35 items divided into seven
sections to measure the minimum quality standards of content,
instructional design, course outline, learning environment,
intellectual property usability and technical aspects.
The second one was based on the Seven Principles for
Good Practice in Undergraduate Education by Chickering
and Gamson (1987). This checklist was developed by the
University team to review blended learning courses, and had
65 items divided in seven sections on a scale of 1 to 5.
The third checklist was developed by Vai and Sosulski
(2011) [8]. This included 87 items related to ten essential
elements of online course design such as learning outcomes
and content, ease of communication, pedagogical and
organizational design, visual design, engaged learning,
collaboration and community, assessment, feedback
evaluation and grading, and ease of access.
As a result of considering the tools, a new checklist
(version A) was developed taking into consideration the
University needs and the scope of work. Table I gives an
overview of the version A of the checklist. It comprised 45
items arranged in 6 categories: 1) Contact information, 2)
Copyrights, Privacy and Security of Information, 3) Nature of
interactive online content,4) Usability, 5) Multimedia and
interface design and 6) Technical specifications. Each item
was measured on a 5-point scale: 1=not at all, 2=minimally,
3=fairly, 4=mostly, 5=completely.
TABLE I: OVERVIEW OF THE CHECKLIST (VERSION A)
Categories Number of items
Contact information 8
Copyrights, Privacy and Security
of Information 4
Nature of interactive online
content 13
Usability 9
Multimedia and interface design 8
Technical specifications 3
The first category included items related to faculty
members‟ contact information, hours when they were
available to the students, the minimum expected response
time to students‟ queries and information about availability of
IT Help desk support for the students. The availability of
information regarding how to contact faculty and support staff
members was meant to help in creating a supportive
environment for students who might be studying in this mode
for the first time. For example, by mentioning the time it
would take a faculty member to respond to their email queries
would help in managing students‟ expectations of faculty‟s
availability during the online teaching.
The second category contained items about the use of
copyright materials on the course website. The items related
to students‟ privacy rights and their data protection on the
course website were included. The University follows a
process of seeking copyright clearance of external materials
that are placed on the course website. Also, there are policies
regarding students‟ data protection. Hence, these were
included in the checklist. For example, “Course materials
contain statements clarifying ownership and usage rights
where appropriate”.
The third category contained items that measure whether or
not online activities promote student-student and
student-teacher interaction. These types of interaction are
essential to ensure student engagement in the learning process.
For example, an item was included: “online activities promote
student-student interaction and collaboration (e.g. use of
discussion forum, wikis, Google docs etc. for group projects,
cooperative learning).”
The fourth and fifth categories measured the ease of use,
multimedia and design quality of the course website,
respectively. They included items related to the design,
navigation and course architecture. For example, these items
were included: “Instructions are clear to complete activities
and assignments, links to various sections on the website are
linked” and “The online interface design has a clear and
logical layout of content.”
The sixth category comprised items related to the hardware
and software requirements for course activities, portability of
the content and also the „Sharable Content Object Reference
Model (SCORM)‟ aspects (SCORM 1.2 complaint). For
example, “the digital media content has been designed to be
accessed on multiple types of devices (portability)”.
B. Validity Assessment
International
Journal of Information and Education Technology, Vol. 6, No. 3, March 2016
225
Face validity was performed to ensure the items and
measures assessed the categories identified in the tool. A
blended learning expert performed the face validity. The
reviewer looked at the checklist, read all the items and agreed
that the items in the checklist appeared to be measuring the
main aspects of the blended learning content design (e.g.
contact information was available, the design of course
website interface is user-friendly, and materials used are
copyrighted).
For content validity, the checklist was sent to three
reviewers, who had experience in designing courses for
blended learning. The reviewers were asked to rate their
opinion on a five-point scale on the "relevancy" („1‟ being
completely irrelevant to „5‟ being completely relevant) of
each item according to their understanding of the essential
components for a blended learning course website. Moreover,
they were asked to comment on various aspects of the tool
(e.g. clarity of language, appropriateness) and give
suggestions to further improve the items in the tool.
All the items were rated between „relevant‟ and
„completely relevant‟ by the three reviewers. The following
changes were made based on the comments from the
reviewers:
1) A few items were re-phrased and examples were added.
For instance, the item “Students are required to post their
contact information online so they may communicate with
each other early on the course” was re-written as “Students
are required to post their contact information early on the
course, so they may communicate with each other.”
2) Three items were deleted from the nature of interactive
online content category. These included items like “the
importance of scheduling/using time wisely is emphasized
during the orientation week/module”; “students‟ initial
knowledge about the use of relevant ICTs is assessed
within the first week of starting the course and/or module”.
These items were related to students‟ orientation
programme and hence were found irrelevant to the course
design.
The revised checklist after the content review (version B)
contained 42 items (Table II).
TABLE II: OVERVIEW OF THE CHECKLIST AFTER CONTENT REVIEW
(VERSION B)
Categories Number of items
Contact information 8
Copyrights, Privacy and Security of
Information 4
Nature of interactive online content 10
Usability 9
Multimedia and interface design 8
Technical specifications 3
C. Reliability Assessment
The tool, after the content review, was sent to two raters to
establish the inter-rater reliability. Both the raters had
knowledge and experience of eLearning design and blended
learning course content development.
The raters were given access to a course site, which was
previously taught through blended learning approaches. They
reviewed the course website using the checklist. The data
from both the reviewers were entered into Statistical Package
for the Social Sciences (SPSS 20.0) to measure the level of
agreement (kappa, ρ) between the two raters.
The results in Table III show a moderate level of agreement
between the two raters and the difference is found to be
statistically significant (ρ = 0.41, p<0.001) [9].
Due to the difference in the rating of the two raters, team
met the raters and had a follow-up discussion on their rating of
the items for refinement and finalization.
D. Tool Refinement and Finalization
Based on the results obtained from the reliability analysis
and qualitative remarks obtained by the raters, a discussion
was carried out with them individually. The purpose was to
ascertain their understanding and rationale for their ratings of
the items in the checklist, especially where there were marked
differences between the two raters.
TABLE III: INTERRATER RELIABILITY RESULTS
Value Asymp.
Std. Errora Approx
. Tb Approx.
Sig
Measure of
Agreement
(Kappa) .410 0.089 5.004 .000
No. of valid cases 40
a. Not assuming the null hypothesis.
b. Using the asymptotic standard error assuming the null hypothesis.
The discussion helped decide whether to keep an item as it
was written, modify/rephrase it or delete it completely.
The key findings from the discussion are as follows:
1) A few items needed to be rephrased to make them more
relevant. For example, “The faculty‟s role in reminding/
encouraging students when they do not complete the tasks
on time is clearly articulated” was rephrased as “A
schedule of timelines is provided to clearly guide students
by when events and learning activities should be
accomplished.”
2) Examples were added for clarification. For example, in
one item, APA was added as an example: “all quoted
materials are cited correctly by adhering to one of the
standard citation format (e.g. APA).”
3) Duplicate items were deleted. For example, “Data
protection policies are clearly stated where required” and
“Steps have been taken to protect students' educational
records/privacy rights (e.g. it is a password protected
site)”, were merged as “Course websites are password
protected and only authorized users are provided access to
the site.”
4) The rating scale from a five-point scale was changed to
three categories, namely, „Yes‟, „No‟ and „Not applicable‟
in order to be more precise.
TABLE IV: OVERVIEW OF THE FINAL VERSION OF THE CHECKLIST (VERSION
C)
Categories Number of items
Contact information 7
Copyrights, Privacy and Security of
Information 3
Nature of interactive online content 11
Usability 6
Multimedia and interface design 6
Technical specifications 2
International
Journal of Information and Education Technology, Vol. 6, No. 3, March 2016
226
International
Journal of Information and Education Technology, Vol. 6, No. 3, March 2016
227
During the discussion, it was observed that training faculty
members, the course designers and other stakeholders to use
the checklist is needed in order to ensure uniform
understanding of each item.
The authors finalized the checklist after carefully reviewing
each discussion point (Version C). Table IV gives an
overview of the final version of the checklist. It comprised a
total of 35 items.
IV. CONCLUSION
This paper has described the process of developing a
checklist for quality assurance of the course content that is
used as a part of courses offered through blended learning
approaches. The process and findings highlight the
importance of developing and using tools that are
contextually relevant. The reliability and validity testing
showed the importance of stating the items clearly; adding
examples with items, having less number of items to keep the
overall length of the tool short, and reduce redundancy. Along
with a checklist for the review of the course website, it would
be necessary to establish quality standards for a blended
course that are overcharging and include other aspects of the
teaching and learning process. The checklist will be
implemented in the final quarter of 2014. Future studies will
capture the experiences of various faculty members,
eLearning designers and developers of using the checklist,
and to design quality standards for the entire blended teaching
and learning process.
REFERENCES
[1] V. Diaz and M. Brown. (November 2010). Blended learning: A report
on the ELI focus session. Educause Learning Initiative paper 2.
[Online]. Available:
http://net.educause.edu/ir/library/pdf/ELI3023.pdf
[2] R.A. Ellis and R. A. Calvo, “Minimum indicators to assure quality of
LMS-supported blended learning,” Educational Technology & Society,
vol. 10, no. 2, pp. 60-70, 2007.
[3] T. Chao, T. Saj, and F. Tessier. (January 2006). Establishing a quality
review for online courses. The Educause Review. [Online]. Available:
http://www.educause.edu/ero/article/establishing-quality-review-onlin
e-courses
[4] P. Ratnasingam, “Quality indicators that impact the design and
delivery of online instruction,” Journal of Contemporary Issues in
Business Research Volume, vol. 3, no. 1, pp. 19-34, 2014.
[5] C. Ceobanu and L. Asansului, “A theoretical framework for quality
indicators in elearning,” Turkish Online Journal of Distance
Education, vol. 10, pp. 126-135, October 2009.
[6] A. W. Chickering and Z. F. Gamson. (1987). Seven principles for good
practice in undergraduate education. Washington Center News.
[Online]. Available:
http://www.lonestar.edu/multimedia/SevenPrinciples.pdf
[7] D. Dayton and M. M. Vaughn. (November 2007). Developing a quality
assurance process to guide the design and assessment of online courses.
Technical Communication. [Online]. Available:
http://www.academia.edu/3779012/Developing_a_Quality_Assuranc
e_Process_to_Guide_the_Design_and_Assessment_of_Online_Cours
es
[8] M. Vai and K. Sosulski, Essentials of Online Course Design: A
Standards-Based Guide, New York: Taylor & Francis, 2011.
[9] J. R. Landis and G. G. Koch, “The measurement of observer agreement
for categorical data,” Biometric, vol. 33, pp. 159-174, March 1977.
Azra Naseem has a master degree in elearning from
University of Technology Sydney Australia, and in
linguistics from University of Karachi Pakistan. Her
research and teaching is in the area of elearning and
ICT enhanced teaching in higher education.
She is currently the assistant director blended and
digital learning network, and a faculty member at the
Aga Khan University. Her publications include:
“Collaborative tools for research, development, and
professional service,”Ethnicity and Race: Creating Educational
Opportunities Around the Globe, etc.
Ms. Naseem has received the Best Teacher Award at the Aga Khan
University. She is also the recipient of Commonwealth Youth Leadership
Award in 2005.
Eman Rashwan is a senior instructional designer and
e-learning consultant at Aga Khan University
She has done her bachelors of computers and
information from Cairo University Egypt and diploma
in e-learning from the Information Technology
Institute, Egypt. She holds requirements engineering
certified from the American University, Egypt.
Prior to her current role as a blended and digital
learning specialist, she has worked for 7 years in e-course design and
development for Middle East Universities and Educational institutions.
Sahreen Chauhan has done her master degree in
applied mathematics from the University of Karachi,
Pakistan.
She is the senior research coordinator, blended and
digital learning network at the Aga Khan University.
She manages collaborative research studies conducted
during the faculty development programme in blended
and digital learning. Her research interests include the
use of digital media technologies in education, and
cognitive tests and tools development.
Ms. Chauhan has worked as a teacher educator and a researcher for 8 years
in the same institution.