Available via license: CC BY-NC-ND 3.0
Content may be subject to copyright.
1877-0428 © 2010 Published by Elsevier Ltd.
doi:10.1016/j.sbspro.2010.03.183
Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
A
vailable online at www.sciencedirect.com
WCES-2010
E-Universit
y
:
Critical success factors
Zuhal TanrÕkulu
a
*, Cicek Tugcu
a
, Selda Yilmaz
a
a
Department of Management Information Systems, Bogazici University, Istanbul, 34342, Turkey
Received October 12, 2009; revised December 21, 2009; accepted January 6, 2010
Abstract
In Turkey, e-learning used in university education, especially in diploma programs, is increasing day by day. In this research, a
questionnaire has been submitted to students studying e-learning diploma programs in Turkish universities. Several dimensions
are created for the questionnaire to widen the analysis. The dimensions are demographic characteristics, which state general
features of students, and success factors, which measure the success of e-learning systems, namely usability, interaction,
functionality, reusability, evaluation, appropriateness, design, interoperability, and accessibility. We focused on the current
situation and success of e-learning systems in Turkey with respect to these factors.
Keywords: e-content design; e-learning; e-university; learning management systems (lms); success factors
1. Introduction
Many universities are using e-learning systems with or instead of traditional face to face education in the world.
The world-wide-web is increasingly being used as a resource in education of students in universities. In Turkey, e-
learning used in university education, especially in diploma programs, is increasing day by day. Now, there are
many universities having e-learning systems in Turkey. The general situation of e-learning systems in Turkey with
respect to the following criteria, and the success degrees given to these criteria form the problem statement (Kor &
Tanrikulu, 2008).
Demographic characteristics of the sample are based on gender (male, female), education program type (associate
degree, undergraduate, graduate), faculty name, department name, state of employment (employed, unemployed),
occupational branch type (governmental institution, private sector, self-employment) and whether the student has
the formal educational diploma or not. The “usability” factor denotes the ease of use for a particular tool or other
human-made object in order to achieve a particular goal. It plays an important role toward the success of e-learning
applications, tools, and systems (Ardito et al., 2006). The “interaction” factor is based on the fact that e-learning
does not mean learning entirely in isolation, interacting only with a computer. Building forms of interaction into an
* Zuhal TanrÕkulu. Tel.: 0532 324 6375
E-mail address: tanrikul@boun.edu.tr
© 2010 Elsevier Ltd.
Open access under CC BY-NC-ND license.
Open access under CC BY-NC-ND license.
1254 Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
e-learning tools, either with faculty members, other students or practical experiments that need to be completed
before progressing, should not be viewed as a failure of the technology. The “functionality” factor is based on the
fact that learning technologies can be expanded in functionality in order to serve broader populations and
educational purposes (Kor & Tanrikulu, 2008). The success of e-learning systems can be increased if they can be
leveraged beyond their original scope. The “reusability” factor refers to the ability of a component to function and
integrate outside the environment for which it was primarily designed (Varlami & Apostolakis, 2006). The
“evaluation” factor is systematic determination of significance of something using criteria against a set of standards.
Evaluation is often used to determine subjects of interest in education as any other branches, so it is not questionable
to use evaluation for e-learning systems as well. The “appropriateness” factor in e-learning systems can be taken
into consideration from two different concepts. One side deals with the appropriateness level of the learning content
while the other side deals with the appropriateness level of the student using the e-learning system (Blass & Davis,
2003). The “design” factor is based on the idea that it becomes a barrier to effective learning because a poorly
designed interface makes students spend more time in learning it than in mastering the provided knowledge (Horton
& Horton, 2003). The “interoperability” factor refers to the ability of different systems and organizations to work
together. Systems, tools, or applications should operate together when needed. The “accessibility” factor refers to
the ability to access the systems or products such as a service or environment. It describes the degree to which a
system is accessible by as many people as possible (Kor & Tanrikulu, 2008).
1.1. Research Objectives
The main aim of the research related to the defined problem statement is to find out the general situation of e-
learning systems in Turkey, and to measure the success of e-learning programs with respect to factors, which are
already defined as success factors.
2. Methodology
This research study started in September of 2008 and lasted for approximately three months. An original
questionnaire was prepared, a sample was formed, and data was collected and analyzed according to the success
criteria.
Figure 1. Research Model
Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
1255
Several factors were formed to be analyzed and each factor was examined through the questionnaire. Figure 1
shows the general framework of the research. There are ten main dimensions, namely demographic characteristics,
usability, interaction, functionality, reusability, evaluation, appropriateness, design, interoperability, and
accessibility.
It is important to emphasize that we determined our success ratio for each factor mentioned above as 80 %. Thus,
after the questionnaire hypothesis is analyzed and tested, we will be able to comment on each factor based on the
previously determined ratio. This ratio is called the “success ratio for factors”.
2.1. Data collection
The questionnaires were sent to university students in Turkey who are studying in e-learning diploma programs,
via e-mail, mail and by hand. The number of questionnaire sent to universities was 130. The overall return rate,
which is 77 %, was slightly higher in some universities in contrast to others. Moreover, there was not any return
from some universities. Although the return rate of survey was high, about 20 % of the questionnaires were returned
with too many missing answers. Therefore, the total number of questionnaires that were accepted for our survey
diminished a bit more. The number of questionnaires differs among universities with respect to their e-learning
program numbers. Finally, the overall return rate of the questionnaires without missing answers to the survey is
61%.
The collected data was analyzed by using SPSS version 16.0 as a software tool.
2.2. Sample
The sample of the research is the students who are studying e-learning diploma programs in Turkey. The
questionnaires were submitted to 80 students. While submitting questionnaires to them, we guaranteed to keep all
the information that was gathered confidential. The gender distribution of the students answering the questionnaire
is 32 female students and 48 male students. The working state distribution, which means that the distribution
whether students are employed or not, is that 52 students are employed, meaning having a job, and 28 students are
unemployed, meaning not having any job. The distribution is not highly homogenous, nor mildly homogenous.
According to the test statistics, there is a significant difference between the number of employed students and the
number of unemployed students, with a significant level of 0.007, which is less than 0.05 (Sig < 0.05).
The program type distribution is that 57 students are from associate degree, 21 students from undergraduate
degree, and 2 students from graduate degree programs. The distribution is not very homogeneous.
In total, there are 10 different departments. According to the department type distribution, 18 students are from
“Information Management”, 7 students are from “English Language Teaching”, 11 students are from “Preschool
Teaching”, 2 students are from “Teaching of Children with Growth Disability”, 10 students are from “Computer
Technologies and Programming”, 5 students are from” Management”, 3 students are from “Accounting”, 9 students
are from “Web Technologies and Programming”, 10 students are from “Divinity Undergraduate Completion
Program”, and 5 students are from “Computer Programming” department.
2.3. Reliability and validity analysis
In order to measure the reliability of the survey, the Cronbach’s Alfa Coefficient is determined. The value of
Cronbach’s Alpha Coefficient should be greater than or equal to 0.6 to trust the reliability of the survey. This
measurement is done for all Likert scale questions from 37 items; the Cronbach’s Alpha is 0.897. Therefore, there is
a high internal consistency of reliability.
1256 Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
3. Results
Statements related to the “usability” factor: In the questionnaire, the students were asked whether or not the
e-learning systems they are studying on, included specific criteria that we defined in order to measure the success
degree of e-learning systems with respect to the “usability” factor. Table 1 includes the criteria and the statistical
means of the criteria shows the statistical results of students’ responses.
The statistical results of the responses indicate that e-learning systems in Turkey mostly contain instruction
manuals about the system usage having the highest mean of 0.9250, and they have a deficiency in instruction
manuals about education sources having the lowest mean of 0.2875. Most importantly, the overall “usability” mean
was calculated to measure the current state of e-learning systems in Turkey by summing the mean values of the
criteria shown in Table 1 and dividing the score to the number of criteria. The success degree was found as 0.74.
Table 1. Statistics of current state for usability factor
Statements related to the “interaction” factor: Some criteria were defined in order to measure the success of
the current state in e-learning systems with respect to the “interaction” factor. In the questionnaire, the students were
asked which of the previously stated criteria, namely forum platform, chat/discussion boards, the homework module,
web-mail, the suggestion module, web-questionnaire, videoconference, electronic letter (student letter, transcript,
etc.), and online courses, are included in e-learning systems they are studying on. The existence of the defined
criteria increases the success of the “interaction” factor for e-learning systems.
The statistical results indicate that the “forum platform” has the highest mean with a value of 0.9125 and the
“web-questionnaire” has the lowest mean with a value of 0.1625 among all criteria, which means that most e-
learning systems in Turkey have the “forum platform” but they do not apply the “web-questionnaire”. In addition,
“videoconference” and “online courses” have low mean values as 0.5000 and 0.6000 respectively. The “chat
module/discussion board” has 0.6625, the “homework module” has 0.6375, “web-mail” has 0.7125, the “suggestion
module” has 0.6125, and “electronic letters” has 0.715.
The result of the calculation indicates that e-learning systems in Turkey have 61% of the criteria, which represent
the success for “interaction” factor. However, it is less than the success value we have already defined as 80 %.
Statements related to the “functionality” factor: Some criteria were defined in order to measure the success of
the current state in e-learning systems with respect to the “functionality” factor. In the questionnaire, the students
were asked which of the previously stated criteria, namely the announcement module, academic calendar, course
presentation with animations, vocal presentations and videos, dictionary, games, help menu, and settings menu, are
included in e-learning systems they are studying on. The existence of the defined criteria increases the success of the
“functionality” factor for e-learning systems.
The statistical results indicate that the “announcement module” has the highest mean with a value of 0.9875 and
the “games module” has the lowest mean with a value of 0.00 among all criteria, which means that most e-learning
systems in Turkey have the “announcement module” but they do not have the “games module”. The “academic
calendar” has 0.9750, “course presentation with animations” has 0.8875, “vocal presentations and videos” has
0.8125, “dictionary” has 0.1125, “help menu” has 0.5625, and “setting menu” has 0.5125.
The result of the calculation for the overall mean indicates that e-learning systems in Turkey have 60 % of the
criteria, which represent success for the “functionality” factor. However, it is less than the success value we have
already defined as 80 %.
Statements related to the “reusability” factor: In the questionnaire, the students were asked which of the
criteria in order to measure the success of the current state in e-learning systems with respect to the “reusability”
factor, namely additional course sources, homework subject, exam questions, online courses, prepared homework,
Instruction
manual about
the system
usage
Instruction
manual about
education
sources
Section showing
the aim of the
subjects in
course sources
Search
engine
Frequency
Asked
Questions
section
Technical
service
support
Program
links
Site
map
N Valid
80
80 80 80 80 80 80 80
Missing 0 0 0 0 0 0 0 0
Mean .9250 .2875 .8250 .6625 .9125 .7750 .8625 .7125
Sum 74.00 23.00 66.00 53.00 73.00 62.00 69.00 57.00
Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
1257
answer key, uploaded homework, and uploaded exam answer, are archived in e-learning systems they are studying
on. Archiving the defined criteria increases the success of the “reusability” factor for e-learning systems.
The statistical results indicate that “archived additional course sources” has the highest mean with a value of
0.9250 and “uploaded exam answer” has the lowest mean with a value of 0.4229 among all criteria, which means
that most e-learning systems in Turkey have additional course sources but they do not have uploaded exam answer.
The “archived homework subject” statement has the means of 0.6500, “archived exam questions” has 0.6375,
“archived online courses” has 0.5000, “archived prepared homework” has 0.5375, “archived answer key” has
0.4375, and “uploaded homework format” has 0.4875.
The result of the calculation for the overall mean indicates that e-learning systems in Turkey have 57 % of the
criteria, which represents the success for reusability factor. The overall success degree is 57 %, which is less than
the success value we have already defined as 80 %.
Statements related to the “evaluation” factor: To measure the success degree of e-learning systems with
respect to the “evaluation” factor, the students were asked to specify their participation frequency levels to the
statements related to the tools and medium in the e-learning since the participation level will show the success of the
system. Statements are as, “I read tutors’ comments in every phase of my projects”, “I read tutors’ comments on my
exam results, and homework”, “I evaluate myself by using online and offline exams, tests, homework, and other
measures”, and “I check my current grade information in the system”.
According to the statistical results of the responses indicate that “I check my current grade information in the
system” statement has the most frequent participation level with the highest mean of 3.9, and “I read tutors’
comments in every phase of my projects” statement has the least frequent participation level with the lowest mean of
2.7250. “I read tutors’ comments on my exam results and homework” statement has the mean of 3.0375, and “I
evaluate myself by using online and offline exams, tests, homework, and other measures” has the mean of 3.6625.
To be able to evaluate the overall success degree of the “evaluation” factor in e-learning systems, participation
levels for all statement are taken into account as a whole. The results indicate that e-learning systems in Turkey are
not successful because of the fact that the overall success degree for “evaluation” factor is 66 %, which is less than
the success value we have already defined as 80 %.
Statements related to the “appropriateness” factor: To measure the success degree of e-learning systems
with respect to the “appropriateness” factor, the students were asked to specify their participation levels to the
statements related to the “appropriates”.
According to statistics, “My department in e-learning system does not require any handcraft and practical
application specialties” and “I have the necessary technical equipment to use the e-learning system” arguments are
the most successful evaluation criteria with the mean of 4.2750. “Education quality of the e-learning system I study
on is high” argument is the least successful evaluation criterion with the lowest mean as 2.8750. In addition, it is
important to emphasize that “The contents of courses in the system are appropriate for e-learning applications” is
another successful criterion, with the value of 4.0875, which increases the appropriateness of e-learning systems in
Turkey. “The e-learning system I study on has a robust infrastructure” statement has the means of 3.0250. The
overall success degree is 74 %, which is less than the success value we have already defined as 80 %.
Statements related to the “design” factor: The success degree of the e-learning systems is measured with
respect to the “design” factor. To measure the success degree, the students were asked to weight the criteria, which
e-learning systems they are studying on include, according to Likert scale.
The statements related to the “design” factor are as “The language in the system is clear and easy to understand”,
the “The format, size, color, and place of buttons in different pages of the system are consistent”, “I can easily
transit on pages by using menus in the system”, “The text amount and quality enable me to read writings easily”,
and “Multimedia mediums such as audio, graphics, animations, and video direct me to learn without distracting my
attention”.
The statistical results of the responses that “The text amount and quality enable me to read writings easily”
argument is found as the most successful evaluation criterion with the highest mean of 3.85 and “I can easily transit
on pages by using menus in the system” argument is found as the least successful evaluation criterion with the
lowest mean of 3.50. “The language in the system is clear and easy to understand” argument has 3.75, “The format,
size, color, and place of buttons in different pages of the system are consistent” has 3.55, and “Multimedia mediums
such as audio, graphics, animations, and video direct me to learn without distracting my attention” has 3.6250.
The overall success degree for design factor is 73 % that is very close to the success value we have already
defined as 80 %.
1258 Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
Statements related to the “interoperability” factor: In the questionnaire, the students were asked whether or
not the e-learning systems they are studying on, include specific statements which we defined in order to measure
the current state of e-learning systems with respect to the “interoperability” factor. These statements are as “opening
the system web pages from different browsers without any problem”, “watching the videos which are in the system,
on every medium (individual and other computers)”, “watching the animations which are in the system, on every
medium”, “seeing the pictures which are in the system clearly, on every medium”, and “running the audio files in
the system without any problem, on every medium”. The existence of these criteria provides the success of the
“interoperability” factor in e-learning systems.
The statistical results of the responses indicate that e-learning systems in Turkey mostly have the opportunity to
see the pictures which are in the system clearly, on every medium which has the highest mean of 0.6125, and they
have a deficiency in watching video on every medium, which has the lowest mean of 0.4500. “Opening the system
web pages from different browsers without any problem” has the means of 0.5625, “watching the animations which
are in the system, on every medium” has 0.5875, and “running the audio files in the system without any problem, on
every medium.” has 0.5375. The overall “interoperability” mean is calculated as 0.55, which is less than the success
value we have already defined as 80 %.
Statements related to the “accessibility” factor: Following criteria were defined in order to measure the
success of the current state in e-learning systems with respect to the “accessibility” factor. In the questionnaire, the
students were asked which of the previously stated criteria they could perform in e-learning systems. Performing the
defined criteria increases the success of the “accessibility” factor for e-learning systems.
According to the statistical results of the responses indicate that the “opportunity to see the course plan and
schedule” statement has 0.9625, “opportunity to see the definition of the courses” has 0.9125, “opportunity to access
the content of the courses on a weekly basis” has 0.9625, “opportunity to reach the sources tutors uploaded into the
system at anytime” has 0.9375, “opportunity to see the academic calendar information” has 0.7375, “opportunity to
see the exam grades and school report” has 0.6750, “opportunity to see the final exam information” has 0.3000,
“opportunity to read the tutors’ comments about exam grades or homework” has 0.6000, “opportunity to read the
tutors’ comments at every phase of the project” has 0.5125, “opportunity to see my detailed personal information”
has 0.8250, “opportunity to reach the archives in the system at anytime” has 0.7375, and “opportunity to read the e-
mails” has 0.7625.
The statistical results indicate that “opportunity to see plan and schedule” and “opportunity to see weekly based
content” have the highest mean with a value of 0.9625. “Opportunity to see final exam information” has the lowest
mean with a value of 0.3000 among all criteria. This means that most e-learning systems in Turkey give the
opportunity to see plan and schedule, and weekly-based content but they do give the opportunity to see final exam
information. The overall success degree is 75 %, which is less than the success value of 80 %.
4. Conclusion
This study has provided a detailed overview of the key factors in evaluation of e-learning systems’ success in
Turkey. In accordance with the factors that we defined as usability, accessibility, reusability, functionality, design,
interoperability, interaction, appropriateness, and evaluation, a questionnaire was conducted to determine the
success ratio of universities’ e-learning systems. Therefore, this article has presented the results of research aimed at
defining a methodology for evaluating e-learning systems. This methodology combines a study of literature and user
testing for the evaluation of software systems. As mentioned before according to the used methodology, which
requires the definition of specific criteria, success factors that capture e-learning system features have also been
proposed.
In this current phase of the reported research, attention was focused on evaluating the success of e-learning
systems according to the students’ standpoints. In order to perform a deeper evaluation, features concerning
Zuhal Tanrıkulu et al. / Procedia Social and Behavioral Sciences 2 (2010) 1253–1259
1259
effectiveness of the e-learning systems with respect to tutors and administrators’ standpoints should be considered.
The next phase of the research will be conducted to understand tutors and administrators’ standpoints.
Acknowledgement
This study is a part of “Determination of Critical Success Factors, and Evaluation of the E-Learning Systems”
project, which is supported by Bogazici University Research Fund (Project Code: 07HN302). Project is coordinated
by Assoc. Prof. Dr. Zuhal Tanrikulu.
References
Ardito, C., Costabile, M. F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V. (2006). An approach to usability evaluation of
e-learning applications. Journal of Universal Access in the Information Society, 4, 270–283.
Blass, E., & Davis, A. (2003). Building on Solid Foundations: establishing criteria for e-learning development. Journal of Further and Higher
Education, 27(3), 227-245 (19).
Horton, W. & Horton, K. (2003). E-learning Tools and Technologies. Indianapolis: Wiley Publishing, Inc.
Kor, B. & Tanrikulu, Z. (2008). Evaluation of E-Learning Systems: Standards and Choosing Test Tools. Proceedings of 2
nd
International Future-
Learning Conference on Innovations in Learning for the Future 2008: e-Learning, (pp. 764-771), Istanbul: IU Publication.
Varlami, I. & Apostolakis, I. (2006). The Present and Future of Standards for E-Learning Technologies, Interdisciplinary Journal of Knowledge
and Learning Objects, 2, 60-76.
