ArticlePDF Available

A Model for Developing High-Quality Online Courses: Integrating a Systems Approach with Learning Theory A MODEL FOR DEVELOPING HIGH-QUALITY ONLINE COURSES: INTEGRATING A SYSTEMS APPROACH WITH LEARNING THEORY

Authors:

Abstract and Figures

As the demand for online education continues to increase, institutions are faced with developing process models for efficient, high-quality online course development. This paper describes a systems, team-based, approach that centers on an online instructional design theory (Active Mastery Learning) implemented at Colorado State University-Global Campus. CSU-Global Campus is a newly-created online campus within the Colorado State University System, and launches in Fall 2008 with fully-online undergraduate degree completion programs and Master's degrees.
Content may be subject to copyright.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 119
A MODEL FOR DEVELOPING HIGH-QUALITY
ONLINE COURSES: INTEGRATING A SYSTEMS
APPROACH WITH LEARNING THEORY
Maria Puzziferro
Colorado State University-Global Campus
Kaye Shelton
Dallas Baptist University
ABSTRACT
As the demand for online education continues to increase, institutions are faced with developing process
models for efficient, high-quality online course development. This paper describes a systems, team-based,
approach that centers on an online instructional design theory (Active Mastery Learning) implemented at
Colorado State University-Global Campus. CSU-Global Campus is a newly-created online campus within
the Colorado State University System, and launches in Fall 2008 with fully-online undergraduate degree
completion programs and Master’s degrees.
KEYWORDS
Online Course Development, Instructional Design, Online Faculty Development
I. INTRODUCTION AND OVERVIEW
Developing or creating an online course is a highly complex and multifaceted process. There are several
levels of “development” that need to occur, and no one person is likely capable of discharging all of the
expertise levels and roles inherent in the process. In order to meet the growing demand for quality online
education, a course development model that provides a common framework for consistency, design,
pedagogy and content can be very effective.
The notion of collaborative course development is well-described in the literature, in the context of team
dynamics, instructional design, and course quality [1, 2, 3, 4, 5]. Institutions such as Park University,
which employs a standardized online course development model incorporating collaborative efforts
among faculty and instructional design staff, have reported positive results [6]. The model presented in
this paper also uses an integrated, collaborative model for development, but is aligned with Active
Mastery Learning, an instructional development theory being refined and tested at CSU-Global Campus.
We begin from the assumption that collaboration provides an optimal environment for developing rich,
dynamic, and interactive online courses. In addition, we acknowledge the need for a sustainable “business
model” for online course development that offers a scalable production process that is the foundation for
quality, efficiency, and productivity for the entire institution. However, online courses are not widgets;
rather, they are learning experiences, and so the proposed model incorporates learning benchmarks and
best practices for online pedagogy, conceptually aligned with Bloom’s Taxonomy and practical
implementation of Quality Matters criteria [7]. Creating the team culture, defining the learning vision and
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
120 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
framework, identifying the resources, and crafting the production workflow for effective teamwork are all
critical planning elements for administrative leadership.
Worth noting is that the degree of collaboration described in this model does require significant
institutional commitment in terms of resources, staff, technological tools, and overall effort. However,
institutions can adapt this model in multiple ways; for example by appropriately scaling the size of teams,
the use of part-time staff or dual roles to fulfill specific team roles, or allowing longer course production
times.
This paper will examine the collaborative online course development process from an administrative
perspective, and present a systemized approach to developing, implementing and managing high-quality,
streamlined online course production.
II. THE BUSINESS MODEL
Online course production does not occur in a vacuum. The system, processes, and workflow are driven by
the “business functions” of the institution, which include consideration of the research, the market
demand, the market expectations, and the course distribution plan—and building in the flexibility to adapt
to market, technology, and fluctuations. As Greenberg [8] powerfully points out, universities are indeed
businesses, and if they are to compete in the ever-growing competitive online higher education market,
they need to take a hard look at their culture and practices. This must occur at all levels, and most
importantly, in the instructional realm, where semesters are no longer 16 weeks, faculty are no longer the
only “experts,” and the classroom is no longer time and place-bound.
In order to compete in the online higher education market, quite simply, institutions need to get relevant,
high-quality product to market quickly. It is no longer feasible to have 1824 months course production
time, and to leave the format, navigation, and look and feel of the final course product to the discretion of
each individual course developer. In a market where students expect fast service, consistency, and quality,
a more streamlined production approach is now required.
High-quality online course development and delivery are essential goals to the fundamental mission of
CSU-Global Campus. As such, the investment in quality, faculty support, and instructional development
staffing enable the model described in this paper. The most recent Sloan-C report indicates that
approximately three-quarters of respondents from public instructions reported that online education is
critical to the long-term strategy of their institutions [9]. The integration of online education into the
institutional strategic planning and goal-setting process is critical to achieving the level of quality needed
to complete in the competitive industry of online education.
It is also important to emphasize that course developers are customers, too—and this is has become a
competitive market. In a market where there are abundant job opportunities for online course developers
and faculty, institutions must retain the best workforce to be become and remain competitive while
providing quality online course materials. Faculty and other subject matter experts (SMEs) have high
expectations about the process, support, expectations, and workload involved in online course production.
The online course production process must take into account the distribution plan; in other words, to
whom and where will the courses be distributed? Courses are being developed for delivery to a global
audience, and quite possibly, in high volume. Even institutions that are tentatively venturing into the
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 121
online realm, building a process that can handle volume will save time and effort later and support
scalability.
The course production framework must also be flexible enough to adapt to changes in technology, student
and faculty evolving expectations, new research in the field of online pedagogy, and curricular changes.
For this reason, we recommend a concurrent course revision and maintenance framework that constantly
compels the production team to quantitatively and qualitatively assess the efficiency of the framework,
and maintain the flexibility to make adjustments as needed.
In sum, the business model of course production is a framework, but should be flexible and dynamic.
Figure 1: Online Course Production Framework: Guiding Questions
III. DEVELOPING THE QUALITY VISION
AND COURSE DESIGN STANDARD
It would be naïve to propose a production process such as this without considering the political and
cultural context of higher education. Academic freedom and faculty governance structures create a
Who is the market
and what do they
want?
What is the course
delivery/distribution
plan and how does
the production
framework support
that plan?
What are the most
current
technologies, and
how can they
support learning?
What is the process
for course
maintenance and
revision?
What does the
research tell us
about quality
criteria in online
education?
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
122 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
cultural framework that necessitates some adjustment of the processes outlined in this paper. However, all
can likely agree that academic quality is still at the heart of what we do. And, one of the primary jobs of
the online education administrator is to develop a working relationship with faculty based on a mutual
recognition and respect for the not necessarily mutually exclusive guiding principles of course
consistency and academic freedom.
At the very core of “quality” is the principle that pedagogy must be the driver of the production process,
not technology. Although technology is certainly an important consideration, it should always support the
learning goals and objectives at both the program and course levels. There is general agreement of this
principle in the literature [1, 2, 10]; however, in practice the technology can often be very alluring to
creative and innovative instructional development teams--but technological bells and whistles will not
replace good pedagogy. Figure 1 outlines the guiding questions for the process as a whole.
Before the production framework and process can begin, it is critical to have a vision of quality and a
course design standard derived from this vision. The standard must be documented in such a way as to
clearly communicate to the course production team the design expectations and the specifications for the
deliverables. Documentation can include a sample course, a spreadsheet of course elements, and/or a
rubric. Often, we get bogged down in rubrics, technical specifications, and checklists and lose the
“vision” for quality. To help the team stay focused on that vision, we recommend having a quality vision
statement that is used to guide the team, such as the one below:
Quality online courses are well-organized into learning units; have clear learning goals and
objectives; include materials and activities that directly support the learning goals and
objectives; engage the learner through interaction with content, other students and the instructor;
and offer rich and relevant resources for students. Most of all, online courses should be fun,
engaging, pedagogically sound, and relevant.
The vision statement should be grounded in theory, and clearly defined in an operational, as well as
conceptual way. Our learning model is Active Mastery Learning, which begins from the assumption that
the course to be developed is not just flat, one-dimensional “content”—it is a series of learning
environments and activities. Effective learning motivates learners, engages them through communication
with each other and the instructor, develops their skills, and enables learners to transfer their new skills to
other settings. The classic Seven Principles of Good Practice, adapted to technology illustrate the
principles of quality in education [11]:
1. Good Practice Encourages Contact Between Students and Faculty
2. Good Practice Develops Reciprocity and Cooperation Among Students
3. Good Practice Uses Active Learning Techniques
4. Good Practice Gives Prompt Feedback
5. Good Practice Emphasizes Time on Task
6. Good Practice Communicates High Expectations
7. Good Practice Respects Diverse Talents and Ways of Learning
Virtually all educators agree that the Seven Principles of Good Practice represent an excellent starting
framework for developing a vision of quality. We begin the Active Mastery Learning foundation here
also, and emphasize the guiding belief that all students can learn when the learning environment is active
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 123
and authentic. Learners must be engaged in the experience, not just the exercise, of learning.
Thus, authentic learning is an important part of the vision of quality for the course, and emphasizes the
importance of creating course activities that actively engage students in exploring, discussing and
analyzing abstract concepts in real-world, relevant contexts [12]. Lombardi [13] provides an excellent
overview of authentic learning, its importance to today’s students, and examples of specific exercises and
strategies for building authentic learning into online course environments. These activities include
simulations, student-created media, inquiry-based learning, and peer-based evaluation.
We believe that constructivist, active and authentic activities are most effective, and the general
agreement in online education is that “active” and community-based virtual environments are the most
favorable for both student satisfaction and learning outcomes. However, students must also master
content in order to meet certain competencies.
Bloom’s taxonomy explains learning as a progression from simple learning to the higher levels of critical
thinking, as illustrated in Figure 2 [14]. Mastery learning can be conceived of as an instructional
philosophy—at the core is the belief that all students can learn, if given the time and opportunity. Not all
students move up the ladder of Bloom’s Taxonomy at the same pace, but the core belief is that they can.
Mastery learning also does not focus on the content, but focuses on the process of mastering the specific
learning objectives tied to the content.
However, mastery learning is just one piece of the puzzle. In a recent study of the effectiveness of
mastery learning, Kazu, et al. [15] found that mastery learning increased student achievement, but that
without activities that encouraged participation, students grew bored and thus achievement was lowered.
The pedagogical philosophy at CSU-Global Campus is that skills and competencies are just one part of
the learning process. Mastery learning activities that are competency-based provide content mastery
opportunities and immediate feedback, and account for a nominal portion of each course grade. Students
need to become competent in specific skills and topical areas, but then they move on to exercise higher-
level skills such as critical-thinking, peer collaboration, synthesis, evaluation and new knowledge
construction—which all account for 90 percent or more of the course grade in various projects,
assignments and portfolio-quality work. In a global, competitive world, these higher-order skills may give
students the edge over their peers who just have “skills,” and equip them with the proficiencies to be
effective leaders and managers in their field.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
124 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
Figure 2: Levels of Learning, Based on Bloom’s Taxonomy
Derived from Bloom’s taxonomy [14], students learn by:
a) mastering information, such as key concepts, terms and ideas through exposure to the information
and recall,
b) progressively grasping the information by practicing learning activities that focus on recall, trial-
and-error, and building the vocabulary and comprehension of the information,
c) applying the information to a problem-based situation that can be collaborative,
d) analyzing the problem further by deconstructing the information and reconstructing it into a
solution,
e) by applying and analyzing the information, begin to recognize the patterns or relationships
between the information and the problem,
f) creating new knowledge and the ability to reason about the information and apply it practically to
situations.
Thus, as illustrated in Figure 3, our online learning model utilizes instructivist techniques, such as
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 125
mastery learning, to guide students to fully grasp terms/concepts (content), and then employs
constructivist techniques to engage students to apply the information in collaborative contexts. The
expectation is that all learning activities foster the highest degree of instructor-student, student-content,
and student-student interaction, with consideration of the particular discipline and course objectives.
Figure 3: Active Mastery Learning Model
The measurement of quality is a complex and sometimes nebulous task. The Quality Matters program [7]
is a research-based program that has received national recognition for their research-based rubric and
inter-institutional peer-review processes (www.qualitymatters.org). The Quality Matters rubric is an
excellent guide for creating a design standard, or an evaluation system, and can be customized and
adapted as needed. We have adapted the rubric to convey the online course design elements to support
Active Mastery Learning, and the specific quality criteria within each element.
IV. THE COURSE DEVELOPMENT PROCESS:
DEFINING THE TEAM ROLES
While in the early stages of online education, instructors created their own course materials with little
instructional design support; however, tales of frustration and lack of support rang in the halls of
Academe. Faculty are experts in their discipline, having studied and researched extensively; they cannot
be expected to become instructional design experts in the amount of time provided for course
development. Designing an online course requires a systemic process that dissects the course learning
objectives, presents content, interactivity, and assessment.
One overarching goal of this course development model is to provide ample instructional design, media
development, and other resources and support. As Oblinger and Hawkins [16] point out, online courses
are no longer content-driven; rather they are complex, technologically-mediated learning experiences that
require high-level instructional design, multimedia expertise, and technology skills that few faculty
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
126 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
possess. Courses designed under this model reflect the innovative and creative intent of the faculty
content expert, while the instructional design team ensures that each course is Section 508 compliant,
accurate, error-free, tested for usability and incorporates an appropriate balance of content mastery
activities, interactive activities to foster peer engagement, critical thinking assignments, and high-level
synthesis projects.
A. Team and Systems Approach to Online Course Development
Moore and Kearsley [17] in describing the systemic model of distance education, compared it to the
system of the human body. Each part of the body plays an integral part in how the body works as a whole,
some are more important than others, but it takes all parts to support a successful system. The same could
be said for the online course development process which is heavily interrelated to online course delivery
and is a subsystem of the overall distance education system. Peters [18] recognized early on that the
system for distance education was similar to that of the industrialization process because it requires
"careful prior planning on a division of labor basis, costly development, and objectivization through
media" (12, p. 112). While Peters saw this as a comprehensive system for the production of goods
relevant to distance education, the model still applies to the development and scalability of online courses.
Peters' [19] theory focuses on the division of labor for the process and concludes the following rationales:
The development of distance study courses is just as important as the preparatory work taking
place prior to the production process.
The effectiveness of the teaching process is particularly dependent on planning and organization.
Courses must be formalized and expectations from students standardized.
The teaching process is largely objectified.
The functions of academics teaching at a distance have changed considerably vis-a-vis university
teachers in conventional teaching.
Distance study can only be economical with a concentration of the available resources and a
centralized administration. (p. 110)
There is a body of recent literature that supports the notion of team-based online course production. Xu
and Morris [14] studied team roles and curricular decisions of a course development team, consisting of a
project coordinator, a web instructional designer, and four faculty members. They found that despite some
issues arising from the collaborative process, such as increased workload and conflicting opinions on
course materials, the faculty members considered the experience highly positive.
In our model, we recommend a single faculty member working with the instructional development
support team to minimize the potential academic conflict, and also recommend a stronger leadership role
for the Instructional Technologist. Hawkes and Coldeway [4] caution against the “lone ranger” approach,
where faculty are the graphic artist, the web page designer, the instructional designer, and even the
programmer. However, in our model, faculty unequivocally drive the subject matter and work closely
with the instructional technologist on pedagogical decisions, and are solidly supported by the instructional
development team in matters of technology and programming. An important theme in the literature is that
faculty do not resist technology or online course pedagogy when provided with resources and support [5].
The support investment is well worth it to create efficient processes where all team members are utilizing
their specific talents and expertise, and where all team members feel supported.
We suggest the following roles for a team approach to online course production:
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 127
Team Member
Role
Course Developer
Senior subject matter expert/lead faculty member
Ensures that the development is in line with the curricular
goals and ensures academic quality exists
Course Development Team Member
Also a subject matter expert
One or more may participate, depending upon scope of
project, or other special circumstances
Works collaboratively with the lead Faculty developer to
select and/or design course materials, etc.
Instructional Technologist
Acts as a pedagogical and design consultant on up to 10
concurrent development projects
Employs an instructional design model, such as ADDIE
[19]
Acts as a project manager for each development project
Assists the team in selecting course materials and learning
objects that support online pedagogy
Ensures that the course design is adhering to the quality
vision and design standard
Course Technicians (Techs)
Acts as technical support on several concurrent
development projects
Provide programming support and assistance with
technical aspects of the course; for example, load course
content, design course banner and graphics, work on look
and feel of course, find or create learning objects, load
quiz and exam questions into the CMS, help developers
record audio/video, help developers ensure that the
technologies are appropriate for the course, include text
scripts for ADA compliance, test the course links,
proofread, etc.
Librarian / Copyright Clearance
Coordinator Provided through library services for review of courses for
TEACH Act compliance and to handle copyright
clearance, reserve readings, etc.
In the organizational model being established at Colorado State University-Global Campus, the
Instructional Development Unit houses the above team members, and also includes supplementary roles,
including a Director of Faculty Development and Support, a Director of Institutional Effectiveness and
Quality Assurance, Program Chairs, and Core Faculty. Together, the Instructional Development team
supports the entire online course development model and integrates faculty into the organization as key
players and decision makers, not just clients.
1. Who Does What?
The table below illustrates the content elements of an online course, derived from Active Mastery
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
128 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
Learning and the Quality Matters rubric [7], and denotes the team member with primary responsibility for
each online course element.
Course
Element
Standards
Roles
Examples
Course
Overview and
Introduction /
Student
Orientation
Introduction to the course
structure, style, learning
experience, technology
requirements
Explanation of support
resources
Explanation of course
policies
Navigational instructions
Explanation of general
expectations
Instructor introduction
Course Developer and
Instructional
Technologist
Instructional
Technologist to provide
any static institutional
information (such as
support resources,
contact information,
institutional policies,
etc.)
Welcome video
“Icebreaker” activities
Text announcement
Any"housekeeping
items" covered
Syllabus
Detailed syllabus document,
with timelines, course
policies, unit schedules, and
other specification included
in a template provided to
faculty
Instructional
Technologist to provide
template
Course Developer
modifies course-
specific information
Course
Curriculum/
Objectives and
Learning
Outcomes
Course topics are clearly
explicated
Course level learning goals
and objectives clearly stated
and address content mastery,
critical thinking skills, and
core learning skills
Course outcomes are clear
and measurable
Each module and unit
includes this element
Course Developer
Instructional
Technologist
List of clear objectives
and learning outcomes
Identification of
knowledge needed in
subject area to gain
expertise
Course Content
The course materials support
the learning goals and
objectives
The course materials appeal
to diverse learning styles and
are appropriate to the online
learning environment
The course materials are
well-organized into a logical
flow, and easily accessible
All course materials are
appropriately cited and
copyright compliant
Course Developer
Instructional
Technologist and
Course Technician (to
help select types and
formats of content)
Librarian
Textbook readings,
audio clips, video clips,
lecture notes, podcasts
Slideshows
Worksheets for lecture
support
Mastery
Activities are included that
Course Developer
Flash-based learning
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 129
Activities
allow students to practice
applying the informational
concepts presented in the
course content
Activities are non-
threatening, and are
engaging, interesting, and
even entertaining
Activities allow for multiple
attempts
Activities are based on
course materials that are tied
directly to learning
objectives
Activities provide specific,
immediate feedback and
students are able to track
their progress
Course Technician (to
help select types and
formats of content, or
create/load activities)
Librarian (when
necessary, to clear
copyright for learning
objects)
exercises such as games
like Jeopardy,
crossword puzzles, and
digital flashcards
Online quizzes
Simulations
Interaction and
Collaboration Interactive activities promote
interaction among student,
and with instructor
Interactive activities are tied
directly to learning
objectives
Interactive activities engage
students with the application
of the information in a
meaningful and interactive
way in order that they
construct new knowledge in
a collaborative environment
(constructivism)
Rubrics are provided that
specify the interaction
criteria for students
The design of the activities
prompts the instructor to be
present, active, and engaged
with the students
Course Developer
Instructional
Technologist (to help
select activities and
discussions that create
engagement,
interaction, reflection)
Discussions
Synchronous
interactions (Chat,
Whiteboard, etc.)
Group activities such as
team site building
Assessment and
Measurement The types of assessments
selected are based on course
materials that are tied
directly to the stated learning
objectives
There are multiple types of
assessments and they are
weighted such that students
have multiple paths to
success
Students receive
Course Developer
Instructional
Technologist
Exams
Written assignments
Portfolios
Assessment should
never be exams only
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
130 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
personalized feedback from
the instructor
For written assessments,
rubrics are provided that
clearly specify success
metrics
Authentic assessment
techniques are incorporated
Course
Technology Tools and media are selected
to support the learning goals
and objectives
Technology is easily
accessible and current
Technology encourages
students to be active learners
Technology selection poses
no undue financial burden on
students (unless absolutely
necessary to the course)
Course Technician Wikis
Blogs
Journals
Videoconferencing
Accessibility
Course is ADA compliant Course Technician Checklist/review
Of course, the above course element table can be expanded to include more or less detail, and should be
based on the institutional model for online course quality. However, the objective is to have a
“specifications” sheet, and preferably a model online “demo” course to which the instructional
development team can benchmark their progress.
The question of “standard” or “canned courses” should be addressed here. There are good arguments for
and against standardized courses. The scope and depth of standardization is an institutional question, and
must be addressed in the context of the course distribution model. With higher volume of enrollments, it
can be expected that an institution will have multiple course sections being taught by multiple full time
and adjunct instructors. In addition, students will be moving quickly and continuously through courses in
the program. It is important to minimize student confusion over language and navigation, as well as be
able to preserve the curricular integrity of courses so that course sequences, particularly those involving
prerequisites, are carefully managed.
The argument against standardization is typically one of academic freedom, and the negative effects of
standardization on creativity and innovation. We would argue that there is a middle point, and achieving it
lies within the key involvement of faculty in the online course production and delivery processes. As
described earlier, the model at Colorado State University-Global Campus integrates faculty within the
Instructional Development organization as key players and decision makers. We recommend utilizing
core faculty as lead faculty in the production and delivery of sets of related courses, and empowering
them to update and modify their assigned courses, thereby keeping them relevant, “fresh,” and reflective
of new information and research, and new pedagogical techniques.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 131
V. ORGANIZING THE WORKFLOW AND ESTABLISHING
DEADLINES
We propose a fourteen-week phased approach to online course production, as shown in Figure 4.Within
each phase occurs various steps and processes, of which we have provided the crucial ones. The
information and process steps we have provided must be adapted to the particular institution’s framework
and policy guidelines, and other steps note mentioned here will emerge as the entire process evolves.
This, however, provides a framework for the individualized process to grow into.
Figure 4: Phases of the Online Course Production Process
A. Phase I Week 0
To begin the course development process, a contract is issued and signed by the content experts at an
initial orientation meeting. This contract defines expectations for both the subject matter expert(s) and the
course development team along with the commitment to follow the 14 week development process as well
as provides information regarding payment for the course. This contract is important, as it also defines the
institutional policies and any intellectual property issues in the development process. Intellectual property
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
132 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
is a complex issue, and the institutional general counsel can and should provide input on the development
of the legal policy.
Next, the dates for meetings and are established along with a discussion on the resources available for
course development such as graphical design, learning object development and copyright support. The
core team (Instructional Technologist, Course Technician, and Copyright Librarian) is then introduced to
the Course Developer and their roles defined. In our model, the primary “manager” of the workflow
process is the Instructional Technologist, and the “co-manager” is the Course Developer. Each needs to
be empowered to execute their roles effectively and so “buy-in” of the roles is critical.
At this stage, the team should also make decisions about their communication processes, expectations,
and formats. Many Course Developers will be remote, and the group must consider the best technologies
for facilitating the work of the team. The new courseshell will then be created in the courseware
management system and the subject matter expert(s) will be provided access (ids and passwords). At this
point, the initial course syllabus and the assessments already in place will be reviewed for online
conversion and the textbook and its resources are examined (for example, does the textbook provide
specific graphics that could be used in the course design?).
A course planning document should be provided which is merely a springboard for the subject matter
expert to sketch out the course topics and assessments. Institutional policies such as a statement that
requires student online participation will be provided for syllabus development and copyright permission
forms provided to request usage if guided by the copyright librarian as well as a Teach Act checklist that
guides online usage of digital materials. The planning document should be derived from the course design
standard, and parallel the language, style, and navigation of the expected final course product.
An excellent resource for this phase is a course developer orientation and a handbook to guide the
process. This documentation will serve as reference material for the entire development team. There are
numerous excellent examples available freely on the web from various institutions. We also recommend
having a “model” course that the faculty can use as a benchmark and emulate in the design and
development processes.
In Weeks 1 and 2, the real work begins. Planning commences with discussions of the conceptualization,
graphical design, and course planning. After reviewing the learning objectives, the team will discuss any
problem areas they see with the content development such as online lab assignments. The development
team will then examine the course template for any necessary modifications and then assessment strategy
will be reviewed. In this phase, we also recommend involving the Copyright Librarian to begin working
with the Course Developer on consideration of materials, and the identification of any potential copyright
issues with regard to the initial thinking about the course materials.
Assessment for online courses should never be based upon exams only. In the Active Mastery Learning
model, we emphasize multiple paths to success, and active learning techniques. Students will submit
several types of assessments which in turn, provide an overall footprint of their work. Online assessment
should include online discussions and may also include journaling, case studies, quizzes and exams, just
to name a few. The course should incorporate a strategy for building an active community; therefore, an
introductory icebreaker exercise may be discussed with different examples provided.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 133
B. Phase II Weeks 3–5
In Phase 2 and by the third week, the team should create the first prototype unit, which includes
placeholders for images, interactions, learning games, interactive media objects, etc. This prototype
lesson should be checked against the learning model, in this case Active Mastery Learning, and the entire
team should agree on the design, the pedagogy, the navigation, the look and feel, and the connection of
the unit to the learning goals and objectives.
In this phase, the first draft of syllabus and course schedule should also be submitted by the Course
Developer, and should contain the following elements: textbook, supplemental readings list, policies and
resources (per syllabus template which will be provided), modularized 8-week course outline with
readings, topics, learning goals and objectives, assignment/assessment plan, and discussion questions for
each unit. The Syllabus template will also include “static” content (such as contact information, student
resources, institutional policies, etc.), and this content should be standardized and maintained by the
Instructional Development Unit staff.
Once the course unit prototype and syllabus delineating the weekly goals, objectives and content, is
approved, weeks 3-5 are where development should gain momentum. An important part of the process is
having milestones and frequently checking progress. Thus, we recommend that the first half of the course
content is provided to the team (four complete modules) that contains lecture materials, learning
objectives and assessment items. The Copyright Librarian should be involved in this review as well, in
order to identify and address and potential copyright issues. This first half of the course should also be
submitted for academic review, to ensure that the academic integrity of the curriculum is supported by the
course design, pedagogical choices, and technology selection.
C. Phase II Weeks 6–8
Review of the first half of the course content may yield feedback to the development team and
adjustments warranted. The Course Developer is consulted for content revisions, while concurrently he
or she is working on the last half of the course content to be submitted. This time in the phase allows for
catch-up if necessary.
In week nine, the Course Developer will begin submitting content for the rest of the course materials. The
Course Technician will make any necessary edits on the first half of the course after review and also work
on developing learning objects, loading exams and quizzes, and creating placeholders for the content. By
week twelve, all content should be loaded into the course template.
D. Phase III Weeks 1213
Weeks twelve and thirteen are spent reviewing the second half of the course modules as well as
proofreading, testing and revising navigation, electronic activities, etc. The final proofreading and editing
should be completed and all hyperlinks and attachments are checked and tested.
E. Phase IV Week 14
The final week of the course development phase is allocated for final approvals from the Course
Developer, Instructional Technologist, and Course Technician. The Copyright Librarian should also
review for any last minute copyright requests. The course is then submitted to the institution's academic
council for final approval, if necessary. A date is determined for deployment into the course schedule. A
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
134 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
post deployment meeting is scheduled for a week or two after the course has been taught. Careful notes
should be taken by the instructor throughout the first semester the course is taught.
VI. COURSE MAINTENANCE
Once courses are entered into a “repository” for the institution, it is important to establish processes and
policies to maintain the currency of the courses, as well as update or adjust the course based on feedback
from students or faculty teaching the course. Shortly after the course is taught for the first time, the team
will review notes kept by the instructor(s) as it was taught. The following items should be discussed and
revisions suggested:
What worked well?
What were the problem areas?
Suggestions for improvement?
Student Feedback?
Revisions will then be made before the next time the course is deployed. A follow-up meeting should
occur to discuss revisions after the course is taught again. At this point, the course should be cycled into a
normal course update process as determined by the institution which could be 2–3 years or sooner if
necessary.
Much of the course maintenance will be driven by the edition cycles of textbook publishers, if traditional
textbooks are used. And, Course Developers who update courses will expect a payment for the work. This
can become chaotic and costly as course and enrollment volume increases.
There are many possible models for maintenance. Again, we refer to the model of integrating faculty into
the organization. In the Core Faculty model, salaried instructors are responsible for selected groups of
courses, and they manage the revision process. However, in institutions that primarily utilize adjunct
faculty or have shared governance structures where faculty governance and evaluation are dispersed
among multiple academic units, the Core Faculty model may not be feasible. However, an appropriate
model and process must be established.
VII. CONCLUSIONS
As Porter [20] states, “An effective curriculum must be well structured, innovative, filled with usable and
appropriate course content, and interesting to a variety of people who take each course and work through
a series of classes(p. 75).
The model presented in this paper is one way of achieving such an effective curriculum. With minor
adjustments and adaptations, this model should help institutions create a process for organizing online
course development into an efficient, coherent, and focused practice. As we continue to learn from our
peers, research online learning, and keep learning from our mistakes our model should evolve. This model
is currently being implemented at CSU-Global Campus, and formalized research to evaluate the model
based on faculty satisfaction, efficiency, quality of design, and student learning outcomes is planned for
2008–2009. Preliminarily, the feedback has been positive and we are looking forward to reporting on the
efficacy of the model and practical lessons learned in subsequent reports.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4 135
Online education has forever transformed higher education, and we are learning that quality is really
about flexibility and the ability to adapt to the changing demands of learners, the new promises of
technology, and the new competitive landscape of higher education. If higher education is to remain
competitive, we must refocus and redesign our paradigms, as well as design business processes that
integrate with quality assurance models.
VIII. REFERENCES
1. Liu, M.C. Developing Courses for Online Delivery: One Strategy. The Technology Source, 2001.
http://technologysource.org/article/developing_courses_for_online_delivery.
2. Byun, H.P, K. Hallett & C. Essex. Supporting Instructors in the Creation of Online Distance
Education Courses: Lessons learned. Educational Technology 40(5): 57–60, 2000.
3. Xu, H. & L. V. Morris. Collaborative Course Development for Online Courses. Innovative Higher
Education 32(1): 35–47, 2005.
4. Hawkes, M. & D. O. Coldeway. An Analysis of Team vs. Faculty-Based Online Course
Development: Implications for Instructional Design. The Quarterly Review of Distance Education
3(4): 431–441, 2002.
5. Alvarez, D.M., K. Blair, E. Monske & A. Wolf. Team Models in Online Course Development: A
Unit-specific Approach. Educational Technology and Society 8(3): 176–186, 2005.
6. Knowles, E. & K. Kalata. A Model for Enhancing Online Course Development. Innovate 4(2):
2007. http://www.innovateonline.info/index.php?view=article&id=456.
7. The Quality Matters Program is sponsored by MarylandOnline, and was supported in part by the
Fund for the Improvement of Postsecondary Education (FIPSE), U.S. Department of Education.
http://www.qualitymatters.org.
8. Greenberg, M. A University is Not a Business (and other Fantasies). Educause Review 39(2): 10–
16, 2004.
9. Allen, E.I. & J. Seaman. Growing by Degrees: Online Education in the United States, 2005.
Needham, MA: The Sloan Consortium, 2005.
10. Palloff, R.M. & K. Pratt. Lessons from the Cyberspace Classroom: The Realities of Online
Teaching. San Francisco, CA: Jossey-Bass, 2001.
11. Chickering, A.W. & S. C. Ehrman. Implementing the Seven Principles: Technology as a Lever.
American Association for Higher Education Bulletin 10: 3–6, 1999.
12. Donovan, M. S., J. D. Bransford & J. W. Pellegrino. How People Learn: Bridging Research and
Practice. Washington, DC: National Academy Press, 1999.
13. Lombardi, M. M. Authentic Learning for the 21st Century: An Overview. Educause Learning
Initiative, Diana G. Oblinger (Ed.): 2007. http://www.educause.edu/ir/library/pdf/ELI3009.pdf.
14. Bloom, B. S. Human Characteristics and School Learning. New York: McGraw Hill Book
Company, 1976.
15. Kazu, I. Y., H. Kazu & O. Ozdemir. The Effects of Mastery Learning Model on the Success of the
Students who Attended Usage of Basic Information Technologies Course. Educational Technology
& Society 8(4): 233–243, 2005.
16. Oblinger, D. & B. Hawkins. The Myth about Online Course Development. Educause Review
41(1): 14–15, January/February, 2006.
17. Moore, M. & G. Kearsley. Distance Education: A Systems View (2nd ed.). Belmont, CA: Thomson
Wadsworth, 2005.
18. Peters, O. Distance Teaching and Industrial Production: A Comparative Interpretation in Outline.
In D. Sewart, D. Keegan, & B. Holmberg (eds.), Distance Education: International Perspectives,
95-111. London: CroomHelm/St. Martin's Press, 1988.
A Model for Developing High-Quality Online Courses:
Integrating a Systems Approach with Learning Theory
136 Journal of Asynchronous Learning Networks, Volume 12: Issue 3-4
19. Peters, O. Distance Education and Industrial Production: A Comparative Interpretation in Outline
(1967). In D. Keegan (ed.), Otto Peters on Distance Education: The Industrialization of Teaching
and Learning, 107–127. London: Routledge, 1994.
20. Shelton, K. & G. Saltsman. Applying the ADDIE Model to Online Instruction. In L. Tomei (Series
Ed.) & L. Tomei (Vol. Ed.), Advances Series: Vol 2. Adapting Information and Communication
Technologies for Effective Education, 40–57.
21. Porter, L. R. Developing an Online Curriculum: Technologies and Techniques. Hershey, PA:
Information Science Publishing, 2004.
Hershey, PA: Idea Group Publishing, 2007.
IX. ABOUT THE AUTHORS
Maria Puzziferro is currently the Executive Director of Instructional Development for Colorado State
University-Global Campus, where she oversees program and online course development, faculty affairs,
faculty professional development and support, and library services. Prior to this appointment, Maria
served as Associate Dean of Virtual College at Florida Community College at Jacksonville and as
Associate Director of the Information Technologies Institute at New York University. Maria has
extensive experience planning for academic programs, program development, and distance education
operations. She has presented and published extensively on best practices for adjunct faculty recruitment,
screening, hiring, and management, as well as best practices for student success and support in distance
education programs. Maria received a B.A. from Adelphi University in Political Science; M.A. in
Political Science and M.L.S. from St. John’s University and; a Ph.D. in Higher Education from New York
University.
Kaye Shelton is the Dean of Online Education at Dallas Baptist University whose program now offers
twenty-three programs fully online. Teaching online since 1999, Mrs. Shelton holds a certification in
Online Teaching and Learning instruction. Her education includes an M.S. in Education emphasizing
Online Teaching and Learning and is currently pursuing her PhD in Educational Leadership at University
of Nebraska-Lincoln. She has published and presented on developing an online education program,
quality online course development and the best methods for teaching online including a recent book
entitled An Administrator's Guide to Online Education. She has also served as an advisor and consultant
regarding online education programs for several peer institutions.
... Digital interventions in particular can circumvent many of the logistical and financial constraints involved in providing (Mrazek et al., 2019a). Digital interventions also allow for the standardization of key content, thereby ensuring all students receive the same high-quality instruction (Kenney et al., 2004;Puzziferro & Shelton, 2008), while simultaneously having the ability to provide content that is personalized to the abilities, interests, and values of individual students (Dixson, 2010;Wang, 2014). ...
... (Soll et al. 2021,). Advantages of this mode of learning include: (a) flexibility in both location and time, (b) standardization of key content, ensuring participants equally receive high-quality instruction, (c) opportunities for equal participation, and (d) reinforcement of agentic learning due to its accessibility (Clarke et al. 2004;Ebner & Gegenfurtner, 2019;Kaufmann et al., 2021;Puzziferro & Shelton, 2008;Soll et al. 2021). ...
Article
Full-text available
A three-phased study and program development on HIV and Mental Health were conducted for a period of two years. This study encompasses a baseline survey of mental health issues among people living with HIV (PLHIV), the design and development of virtual learning modules on mental health and HIV for health providers, and a pre-implementation qualitative study on the delivery of online modules to target health workers. The results of the quantitative survey reveal moderately high self-reported depression, anxiety, and suicide ideation among patients with HIV. After the survey, topics for e-modules on mental health were identified including self-help modules for health workers. These mental health modules were consolidated with modules pertaining to HIV treatment. A qualitative study on the acceptability and feasibility of the use of the e-modules was conducted by interviewing health workers from different HIV treatment and care sites. Significant insights into the learning experiences among health workers gave researchers a deeper understanding of the cognitive acceptance and resistance towards virtual and self-paced modules.
... These distributions should be involved with the analytics patterns with big data approach. The data can come from online activities where the users usually upload, retrieve and also store the information shared through the Internet platform (Puzziferro and Shelton, 2008;Omar et al. 2011;Hixonet al. 2012). This will lead to fulfil in forming the model pattern of big data which can directly be engaged in supporting the human daily needs. ...
Conference Paper
Full-text available
Nowadays, the Internet connects people, multimedia and physical objects leading to a new-wave of services. This includes learning applications, which require to manage huge and mixed volumes of information coming from Web and social media, smart-cities and Internet of Things nodes. Unfortunately, designing smart e-learning systems able to take advantage of such a complex technological space raises different challenges. In this perspective, this paper introduces a reference architecture for the development of future and big-data-capable e-learning platforms. Also, it showcases how data can be used to enrich the learning process.
... Mentored support, key teaching resources, including tools (e.g., laptop & data) and support outside of business hours, for both faculty and students, would further help faculty successfully transition to online teaching, as would support with course design. With regard to course design, Puzziferro and Shelton [43] proposed a team-based, 14-week phased pre-launch approach to online course production, in which faculty were key team members. ...
Article
Background: The steady growth of online education has resulted in the need for more faculty. Faculty have an integral role in creating the structure, processes and environment for effective student learning and, thus, require preparation and support to effectively perform this new role. As nursing programs expand capacity with online course delivery, the author found limited research on faculty perceptions of preparedness for teaching online. The purpose of this study was to explore faculty perspectives of teaching nursing content online in prelicensure baccalaureate nursing programs. In this article, the focus is on one specific aspect of the study, that is, the data that sought a deeper understanding of how prepared nursing faculty perceived they were and supports they needed for effective online teaching.Methods: The exploratory-descriptive, mixed-methods study design was based on document analysis, an online survey completed by 32 faculty (53.3%) and interviews with 16 faculty in a representative sample of 13 Ontario Colleges.Results: Institutional and faculty supports related to all best teaching practices. More faculty received an orientation to technology compared to the pedagogy of teaching online and experienced some challenges with these supports.Conclusions: Much more time was required for online teaching for which faculty should be compensated in workload assignments. Findings suggest that both technological and pedagogical training be integrated to faculty development programs and faculty be engaged in these programs prior to teaching online.
... Сформулированы конкретные направления профессиональной деятельности педагогического дизайнера, включая управление профессиональной коммуникацией, редактирование методических материалов, развитие университетских медиа, графический дизайн, управление проектами, командообразование, обучение преподавателей и студентов образовательным технологиям [16]. М. Пацифьеро, описывая функционирование команды преподавателя-эксперта и методиста при подготовке цифрового образовательного продукта, задачи методиста определял следующим образом [17]: ...
Article
Full-text available
Introduction. In the article the peculiarities of scientific and communication practices of the methodologists in interaction with the university teachers are considered on the material of the analysis of the ITMO.Expert methodological community activity. Methodology and sources. The theoretical framework of the research is institutional and structural-functional approaches to the study of university activities. The method used for collecting empirical data presented in the article is a content analysis of the co-messages sent by the participants of the online intensive “Personalized Learning Technologies” on the Zoom-conference, which was held on August 23–27, 2021. The research hypotheses are the following: discussions of new pedagogical practices and new design formats initiated by university methodologists provoke teachers' skepticism, distrust, and resistance. Also teachers may demonstrate vulnerability in discussing new approaches in education. Results and discussion. Testing the hypothesis that discussing new pedagogical practices and approaches will cause skeptical attitudes and distrust on the part of teachers was not confirmed. The patterns of distrust accounted for less than 5 % of the total array of messages. The hypothesis of teachers (especially beginners) being vulnerable when discussing new approaches, a manifestation of which was their refusal to discuss their own pedagogical experience and the problems encountered, was partially confirmed. At the same time due to the organizational and communicative actions of the methodologistsorganizers of the intensive there was formed an atmosphere of trust and safety, so the participants had an opportunity to share their experience or ask questions without being judged by their colleagues. Conclusion. The activity of the pedagogical designer is both an educational activity, aimed at teaching and promoting new educational technologies in teaching, and an organizational and managerial activity, aimed at stimulating the activity of teachers in developing and modeling a variety of knowledge formats. Many researchers suggest characterizing pedagogical design as a sociocultural activity. When designing educational experience and new educational products, methodologists help develop pedagogical traditions and reconsider norms.
... First, digital interventions reduce geographical and financial constraints that would otherwise limit equitable access (Devine et al., 2018). Second, digital interventions allow for the standardization of key content, thereby ensuring all students receive the same high-quality instruction (Kenney et al., 2004;Puzziferro & Shelton, 2008). Third, digital interventions can provide content that is personalized to the abilities, interests, and values of individual students, thereby promoting student engagement, cultural responsiveness, and effective learning (Dixson, 2010;Wang, 2014). ...
Article
Educational Design and Evaluation Models are important factors in e-learning as they provide guidance information for proper strategy organization pursuing both specific learning outcomes and ensuring the main elements of e-learning, such as self-regulation and collaborative learning. The examined educational models of ADDIE, Bloom, and Kirkpatrick are widely known and recognized as models for design and measuring the effectiveness of learning in order to achieve the best possible learning outcomes based on the needs of a specific target group in a specific educational context whether traditional or digital. Specifically, the ADDIE Model is a widely known learning design model used by many educational designers and training programmers to develop education and training programs. The Bloom Taxonomy is a method of building learning goals that follows the process of cognition. The Kirkpatrick Model is a method of evaluating the effectiveness of e-learning and educational programs in general, both in terms of training and business performance of learners. The purpose of this paper was both the investigation of the academic performance, the self-regulated learning and the collaborative learning in relation to the models of ADDIE, Kirkpatrick and Bloom in distance online environments and their effectiveness to the learning process. Meta-analysis was applied for research methodology. After a systematic literature review, we found that only 37 articles were appropriate for meta-analysis. Especially, 23 articles were on the ADDIE model, 9 articles were on the Kirkpatrick model and 5 articles were on the Bloom model. According to the results of this study, we found that all models apply to online process and meet different learning requirements. Regarding the cognitive performance of the trainees, all models supported the effectiveness of distance education. Moreover, the self-regulated learning and the collaborative learning, as factors inextricably linked to the effectiveness of the distance education, were examined in a small number studies in the above models. Finally, all three examined models reinforced students’ positive attitudes and perceptions, even while transferring the acquired knowledge to the workplace.
Chapter
Full-text available
The rise of online courses in higher education has necessitated the development of effective assessment strategies to ensure quality learning outcomes. This chapter explores the importance of effective assessment strategies in online courses and provides insights into designing such strategies. It highlights the importance of assessment strategies in online courses, emphasizing their role in evaluating student learning and providing feedback. Various types of assessment strategies are discussed, including their advantages, and disadvantages and the challenges also are examined. The chapter provides guidelines for designing effective assessment strategies in online courses to address the highlighted challenges. By incorporating these guidelines, educators and instructional designers can create assessment strategies that cater to the unique features and challenges of the online learning environment, ultimately enhancing the effectiveness of online courses in higher education.
Article
Full-text available
The matter of building an optimal model of managing the online platform “Higher School Mathematics Teacher” is considered in this article. The existing researches on the management of online platform development and support process have been analyzed in this paper. The model developers described the process of building an online platform according to the Software Development Lifecycle rules. The researchers described an online course platform creation and functioning process that corresponds to a five-stage pedagogical ADDIE model (Analysis, Design, Development, Implementation, and Evaluation) of online course preparation. The research describes the roles and objectives of online platform team members. These results allowed the authors of the article to model the organizational structure of online platform management. The method of survey among the team members of the online platform “Higher School Mathematics Teacher” and its participants was used to prove the efficiency of the offered model. The results allowed us to confirm the actuality of the research in the management of educational platforms, the convenience of teamwork in online platform management, and successful model implementation.
Article
Full-text available
The seven principles of good practice in undergraduate education (Chickering and Gamson, 1987) summarize what research has shown about how to improve student learning (e.g., more intensive faculty-student contact, active learning, high standards, frequent assessment). This article describes how widely available technologies can be used to use (and enhance) each of those seven principles.
Article
The challenges and traits of online teaching are discussed in the 'Developing an Online Curriculum: Technologies and Techniques' book. The book help administrators, teachers, and technology experts to understand each other's perspectives and work together to make online education viable and worthwhile. The book should guide teachers through the process of developing courses, working with learners, and creating support networks. The book will also help teachers to understand the nature of online education and also to make teaching more effective.
Chapter
This chapter assembles best ideas and practices from successful online instructors and recent literature. Suggestions include strategies for online class design, syllabus development, and online class facilitation, which provide successful tips for both new and experienced online instructors. This chapter also incorporates additional ideas, tips, and tricks gathered since the paper was originally published in the October 2004 issues of the International Journal of Instructional Technology and Distance Learning as "Tips and Tricks for Teaching Online: How to Teach Like a Pro!"
Article
This paper represents a review of the second edition of Distance Education: A Systems View by Michael Moore and Greg Kearsley (Thomas/Wadsworth, 2005). This second edition reflects a view of current applications of distance education, based on the vantage of instructional systems design. The strengths of the book are reflected in the uniqueness of its consideration of online courses, in particular from the point of view of systematic development and management.
Article
In order to meet the growing demand for quality online education, Park University has adopted a model that provides a common framework for all of its online courses. Evelyn Knowles and Kathleen Kalata discuss the circumstances leading to the current system and describe the university's implementation of a course development process that ensures quality and consistency in both content and instructional design. In this model, academic departments select which courses they want to have developed into online courses and the subject matter expert who will act as course developer. The course developer works with an instructional designer to create the online course. The course development process is described in detail, from initiating the contract through consultation with the instructional designer to quality standards review. (Contains 5 exhibits and 2 figures.)