ArticlePDF Available

Abstract

A design and application of adaptive online exam system are carried out in this paper. Adaptive exam systems determine different question sets automatically and interactively for each student and measure their competence on a certain area of discipline instead of comparing their gains with each other. Through an adaptive exam technique, a student's distraction and motivation loss that is led by the questions with quite lower hardness level than his/her competency is prevented. In addition, negative effects of questions requiring higher knowledge than his/her competency over a student's self confidence and morale are dismissed. Since questions are specialized so that they can allow making clear deductions about student gains, they are able to detect student competencies more effectively. Requiring less total time for measuring and being more flexible in the exam management are among the advantages provided by the system. Self sufficiency of the system in terms of planning, repeating and assessment of the measurement process especially allows itself to be used in the individual education sets. Through this system, student competencies can be determined more effectively in cases such as distant-learning, in which some challenges are experienced frequently.
P r o c e d i a - S o c i a l a n d B e h a v i o r a l S c i e n c e s 1 1 6 ( 2 0 1 4 ) 3 0 7 9 3 0 8 3
1877-0428 © 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and/or peer-review under responsibility of Academic World Education and Research Center.
doi: 10.1016/j.sbspro.2014.01.711
ScienceDirect
5th World Conference on Educational Sciences - WCES 2013
Designing and implementing an adaptive online examination system
Mustafa Yağci *, Menderes Ünal
Ahi Evran University, Faculty of Education, Kırsehi, Turkey
Abstract
A design and application of adaptive online exam system are carried out in this paper. Adaptive exam systems determine
different question sets automatically and interactively for each student and measure their competence on a certain area of
discipline instead of comparing their gains with each other. Through an adaptive exam technique, a student’s distraction and
motivation loss that is led by the questions with quite lower hardness level than his/her competency is prevented. In addition,
negative effects of questions requiring higher knowledge than his/her competency over a student’s self confidence and morale are
dismissed. Since questions are specialized so that they can allow making clear deductions about student gains, they are able to
detect student competencies more effectively. Requiring less total time for measuring and being more flexible in the exam
management are among the advantages provided by the system. Self sufficiency of the system in terms of planning, repeating and
assessment of the measurement process especially allows itself to be used in the individual education sets. Through this system,
student competencies can be determined more effectively in cases such as distant-learning, in which some challenges are
experienced frequently.
1. Introduction
Almost all of the contemporary testing systems are applied as paper-based. There have been numerous studies to
apply exams online and carry out the measurement and evaluation processes through computers since technological
means such as computer and internet have been widely used in educational activities. In these exams, students take
tests either on a printed document and place their answers on a sheet; or they answer them on a computer screen.
Besides, in these sorts of exams, all students are required to answer the same questions.
Particularly during recent years, artificial intelligence applications toward comprehending human thinking
process and copying this structure within the computer environment have been more prominent. The fuzzy logic,
which is taken as an artificial intelligence at the same time and which also enables modeling of human behaviors
and even expressing the undetermined concepts in mathematical form, has been a method that is applied frequently
with regard to the resolution of decision making problems. Education is one of the main fields where fuzzy logic is
utilized. (Kinshuk et al., 2001). Within the education process, a series of incidents that require decision making can
be encountered. Especially, during measurement and evaluation process, it is a challenging process which requires a
great deal of experiences to make the right decision and evaluate students properly. The issue of evaluating students’
performance without any mistakes has a structure to be modeled through fuzzy logic method because of
indefiniteness and subjective scales contained (Karal & Çebi, 2011).
In this study, the exam system and design stages designed by means of test algorithm in the adaptive exam
system below are explained.
* Mustafa Yağcı (Assist.Prof.Dr.) Tel.: +90-0505-770-1408
E-mail address: mustafayagci06@gmail.com
Keywords:Adaptive exam systems, expert systems, online exam, individualized learning, intelligent eutoring;
Avai labl e on lin e at www.sciencedirect.com
© 2013 The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
Selection and/or peer-review under responsibility of Academic World Education and Research Center.
3080 Mustafa Yağci and Menderes Ünal / Procedia - Social and Behavioral Sciences 116 ( 2014 ) 3079 – 3083
A facility value is assigned to the each question according to a determined fixed scale (e.g. 0-100 or 0-10 or 0-1
etc.). For the question selection, binary-tree technique algorithm is employed.
This algorithm works in the way below:
1- Among the questions in the relevant subject, a question with medium level difficulty is asked first (e.g. if
range of 0-100 is used, a question with 50 difficulty score). If student answers the question correctly, higher
difficulty bracket (50-100 range) is halved to determine the difficulty level of the following question; in our process
the question with level of 75 is asked. If the student answers the question erroneously, the lower difficulty bracket
(0-50 range) is halved and a question with 25 difficulty level is asked.
2- In the successive questions, the difficulty score of the next question is determined by halving the upscale or
downscale threshold value range due to the difficulty level of the latest question.
3- There are essential threshold values required to be used in the algorithm such as:
•The maximum amount of question,
• The minimum amount of question,
• The largest difficulty score difference to let us consider the swinging has slowed down,
• The number of question (usually calculated as a function of standard deviation) needed to be asked to students
to finish the test after the slowdown of the swinging (within the same swinging),
• The interpolation strategy followed after the algorithm stopped (directly related to the threshold specified
above).
4- There must be sufficient amount of questions (on the same difficulty level) prepared to reduce the probability
to face the same question to the acceptable level.
2. Expert systems
The expert systems, searching for how well the computers get human conducted jobs done, is one of sub-
branches of artificial intelligence programming science . An expert system is a computer software relying on
knowledge and deduction to perform a hard task which is supposed to be undertaken by expert people. It has a
database consisting of information specialized in an expertise subject as it is similar to the competency of a person
in a specialization area such as mathematics. Expert people obtain results by making logical deductions based on
their knowledge in their specific area of expertise. Again, expert systems make deductions and reach results based
on their information (Önder, 2003).
Expert system; is a computer software that can perform job in a complex system conducted by an expert person.
These systems also can be referred as computer software with consultancy quality which can perform expertise jobs
requiring analysis, classification and diagnosis (Allahverdi, 2002).
Expert systems can handle irresolute problems by providing expert support and can bring solutions. Due to their
structures, database interface and decision support systems can be employed at the same time (Önder, 2003).
3. Characteristics of the adaptive exam system
Received answers as a result of an adaptive test application can be used in determining the next question or the
question set. While certain amount of correct answers lead to the selection of questions with equal or higher level of
difficulties; similarly, certain quantities of erroneous answers lead to the selection of questions requiring less
competence. Thus, competence and developmental levels of individuals can be determined with fewer questions.
Therefore, by individualizing exams, students are not required to complete whole question sets which are not
suitable to their development level (Wise and Plake, 1987).
Adaptive web-based exam is the process of matching up of measured individual competence (talent) and item
difficulty in the computer environment (Hambleton et al., 1991). In other words, the application is based on
adjusting a test to the individuals’ measured competence level; and on application of different sets of test items due
to an individual’s position in his/her measured characteristics (Weiss, 1988). In this case, each individual receives
different tests with different difficulty levels adjusted to their competence level. Conventional tests are usually
arranged to measure mean competence level. Adaptive test application provides an environment to reveal the best
measurement of an individual’s competence level. A general adaptive test application follows the steps below (Lord
and Stocking, 1988):
1- Competence level of an individual is estimated.
2- Items which can be good measures of that individual’s competence are selected from the pool consisting of
related parameters.
3081
Mustafa Yağci and Menderes Ünal / Procedia - Social and Behavioral Sciences 116 ( 2014 ) 3079 – 3083
3- The selected items are applied and scored, and the estimated competence level is reviewed.
4- If the error in the competence estimation has reached sufficiently low level, the application is terminated; if
not, turned back to the second step.
Exams requiring high level of security must be performed in secure exam rooms under surveillance such as
computer lab. Many specialized test distribution method can be supported by internet today. Web-based exams
allow test distribution and security subjects in the specially determined test centers.
4. Enhanced web-based adaptive exam system
The developed adaptive exam system provides a wide variety of applications to students and lecturers due to its
design and simple visual characteristics which do not require users to have any further guidance. By means of
different methods, exams can be implemented, analyzed; and measurement and evaluation processes can be
performed. In addition to the application of exams through an adaptive way, student monitoring and guidance as
well as assessment of exam results, which is normally neglected due to its time-consuming structure but important in
terms of achieving educational targets, can be more convenient.
4.1. General structure of the system
The system is composed of three-layered structures: a database for data storage, a server for application and
clients which are connecting to the application server. As database, MYSQL is used since;
1- MySQL database management system is open-coded software and it has enhanced specifications,
2- It uses application server layer,
3- On the client layer side, it is required to employ a web browser (Internet Explorer, Mozilla, Opera etc.).
To develop Online Exam system, a server based and fast PHP programming language is preferred.
For the developed software, Apache is used as a web server which is a strong, skilled and flexible HTTP (Hyper
Text Transfer Protocol) server and open-coded programming. The web server is a software sending the pages stored
under the web address you are connected to (Netcraft, 2012).
Macromedia Dreamweaver 8 software is used to prepare the interface of the system. Dreamweaver is one of the
most preferred HTML editors due to its enhanced properties and conveniences; and allows server-based languages
to be used to create dynamic pages.
In the adaptive web-based exam system, JavaScript language is used to allow dynamic user access; to let it
probed in the same page; and to perform tasks such as presenting resting time for the exam.
4.2. Security
To ensure data security, IPSec or Kerberos must be used between database and application server. Since IPSec
provide secure communication between database and application server, it prevents third parties intrusion into the
database. In addition, SSL must be used to assure secure communication between the client and the server. In this
application, securing passwords of administrators and lecturers is of crucial importance. It is quite simple for the
third parties to capture user names and passwords sent over the network without any encryption. Therefore, the data
sent over the network must be encrypted with SSL. Furthermore, as it has been presented in the previous sections,
passwords must be stored as encrypted in database and adequate precautions must be taken against SQL injection
attacks (Akın, 2007).
Some users who are searching for an opportunity attack the system may wish to access and harm the system
through SQL injection commands. In the developed application, PHP and MYSQL commands, such as
HTMLENTITIES, MYSQL_REAL_ESCAPE_STRING, STRIP_TAGS, are used to prevent unauthorized entries
into the system and precautions are taken against those attacks. Additionally, in the inquiries requiring user name
and password, encryption algorithms such as Md5 and SHA1 are used.
When users are granted access into the system, user names, session information are kept and these are checked in
every page requiring access to prevent unauthorized access. Besides these security precautions, users must maintain
their network security at the highest level by means of antivirus and anti trojan programs.
3082 Mustafa Yağci and Menderes Ünal / Procedia - Social and Behavioral Sciences 116 ( 2014 ) 3079 – 3083
4.3. User groups
There are three types of users in the system: system administrator, lecturers and students. System administrator
has the authorization to add new users, courses into the system, assign new lecturers and students to courses, and
add announcements. Lecturers are authorized to define classroom, courses and students, connect the classroom and
the course, add and modify the questions, change student passwords, edit questions, access exam results, include
classroom lists, transform exam results into the excel form for further analysis, and send messages to parties. When
students access into the system, they see the exams they are required to take; they can take these exams; and they
can view their exam scores. User types and roles in this application are presented in the Table 1.
Table 1. User Groups
User Access Authorization Limitations
Administrator To the administrator page Can view all data in the system. All rights granted.
Lecturer To the lecturer page Can view and amend all information
regarding given courses.
Cannot access to the courses given by the other
lecturers.
Student To the student page
Can take the exams of the given courses
and view data about them.
Can only access to the exam information defined for
themselves and to the general student
announcements.
4.4. Organization and setting
In the web-based exam system, the user name and the password are the keys of the system to identify a specific
user. Since any person with the user name and the password is considered to be authorized to access into the system,
the information security is critical. The administrator can automatically generate the user name and the password for
students registered for the course and for the lecturers of the described course; and send this information to parties in
a secure way.
5. Definition and implementation of the exam
An exam can be organized by the lecturers of the course from the question pool that they have already created or
from the previous questions entered into system database. Exams can be prepared in advance but at the same time
students’ access are not allowed. To that end, the exam is performed under IP limitation so that it can be
implemented only in specific computers or laboratories.
Texts, pictures and animations can be used in created questions. The system can be easily accessible twenty -four
seven from any internet access point. Since all information regarding the exam session is transferred to database
through the web interface, lecturers can also reach students’ data through web interface.
The web interface used for the exam system is designed in a way to ensure security by using the least
information level and most user-friendly form. In this application, each exam result can be updated along with the
new information defined. A user manual for the interface applications is a preferred characteristic of the system.
5.1. Creating an exam
Exams can be created by two different methods. In the first method, lecturers can create an exam by selecting as
many as questions they need with a certain difficulty level from the question pool that they previously created by
entering questions with different difficulty levels. While the exam is being created, which group to take it is
predetermined. Thus, students view the exam created for them when they access into the system. On the other hand,
in the second method, the lecturer lists all questions recorded in the question pool for the relevant exam, then selects
the desired questions among them.
5.2. Exam data
For automated evaluation of exams by the system itself, it is also possible that raw data generation process for
analyzing can be performed by using obtained information and exam data. The data are evaluated through statistical
methods and assessments can be made about the current exam system as well as they can constitute resource for
3083
Mustafa Yağci and Menderes Ünal / Procedia - Social and Behavioral Sciences 116 ( 2014 ) 3079 – 3083
future studies. Above all, the selection processes of most suitable questions and dismissal of questions not
accommodating validity, reliability and other analysis criteria from the question pool can be more convenient.
6. Database design
The database infrastructure is carefully emphasized in terms of circulation and recording processes of data
through the web-based exam system. The database is designed to compile the largest amount of data by using
minimum amount of system source while it is keeping all data safe.
Main tables in the system are Courses, Questions, Exams, Students, Lecturers, Administrators, Semesters,
Majors, and Results. Additionally, there are auxiliary tables of Lecturer_Course, Student_Course,
Semester_Courses, Exam_Questions, Student_Answers.
7. Results
Within the developed exam system, an exam can be implemented by means of both the conventional method and
the adaptive exam method. In this study, design and application stages of the exam system are carried out.
Furthermore, the designed exam system was implemented in the midterm and final exams through the conventional
method for the students taking the course of Computer I at the Faculty of Education, Ahi Evran University (Kırşehir,
TURKEY) in spring semester of 2011-2012 academic year.
Framework for the academic success is comprised of 65 items. To enhance the content validity of the test
coverage, proportion of the subjects in the distribution of questions is taken into consideration. Test validity is
provided based on experts’ views. Validity of the tests is ensured based on the pilot study applied to 90 students in
two different classes. As a result of the application, 15 items whose distinctiveness strength is below 0.30 are
dismissed from the academic success test. While the distinctiveness strength of the rest 50 items is in the range of
0.30 and 0.95; their item difficulty varies between 0.27 and 0.64. As a result of the calculations, the mean difficulty
level was 0.52; and content consistency coefficient was (Kr-20) 0.88, and their standard deviation was 4.5. Thus,
difficulty levels and distinctiveness values required for the question bank, essential part of the adaptive exam
system, are determined.
In the previous studies it was concluded that “It is presented that student performance evaluation problems can be
solved by means of multi-criterion decision making methods; and similar methods can be employed in different
measurement and evaluation applications” (Karal and Çebi, 2011).
In further studies, practicability, validity, reliability, advantages and disadvantages of adaptive exam systems will
be discussed through the statistical analyses over data that will be obtained as a result of the future adaptive exam
system applications.
Acknowledgements
The authors would like to express their sincere thanks to Ahi Evran University directors for supporting the study
in the frame of Scientific Research Projects.
References
Akın, O. (2007). Web Tabanlı Sınav Sistemi. Unpublished master’s thesis, Sakarya Üniversitesi, Fen Bilimleri Enstitüsü, Sakarya.
Allahverdi, N. (2002). Uzman Sistemler, Atlas Yayıncılık, İstanbul.
Hambleton, R. K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory. Sage Publications Inc. California.
Karal, H. & Çebi, A. (2011). The evaluation of student performance via the fuzzy logic approach, IETC, Proceedings Book Vol 2, 1445-1451.
Kinshuk, K., Patel, A. & Russell, D. (2001). Intelligent and adaptive systems, Springer, sayı, 79-92.
Lord, F.M. & Stocking, M. L. (1988). Item response theory. In J.P. Keeves (Ed.), Educational research, methodology, and measurement: an
international handbook, Pergamon Press, 269-272. Newyork.
Netcraft Ltd. Anti-phishing and pci security services. Retrieved September 05, 2012, from http://news.netcraft.com.
Önder, H. H. (2003).Uzaktan eğitimde bilgisayar kullanımı ve uzman sistemler. The Turkish online journal of educational technology,2(3),142-6.
Weiss, D. J. (1988). Adaptive testing. In John P. Keeves (Ed.), Educational research, methodology, and measurement: An international
handbook (pp. 372-375). New York: Pergamon press.
Wise, S. L. & Plake, B.S. (1989). Research on the effects of administering tests via computers. Educational measurement: Issues and
Practice, 8(3), 5-10.
... M. Yagci and M. Unal in [4] recently released the design and development of an adaptive exam system. They propose an improved web-based adaptive exam platform for three categories of users: Administrator, Instructor, and Student, with statistical methods and assessments used to test students' data. ...
... The research also suggests certain marking threshold for fair evaluation of descriptive answers based on multiple surveyed parameters. We can infer from [4,5,6,7,8,9,10,11,12,13,14,15] that auto-mated test evaluation is a prerequisite of today's educational systems and that for student assessment, it should be designed on a generic basis, as well as handle security issues and require better performance evaluation algorithms. ...
... Computerized adaptive Tests (CATs) are already present that change the difficulty of questions based on students' responses. Adaptive assessment systems are getting popular as they can challenge the students' learning and test the epistemological growth of students [9]. ...
Chapter
Adaptivity is actively pursued for testing in education as it provides alternative paths of action for the students. Particularly in engineering education adaptive assessments provide an estimation of students’ competence to provide novel solutions and accurate evaluation of knowledge. This chapter discusses the question identification based on Bloom’s taxonomy for the design of Computerized adaptive assessment. Indicators selected for classification are pre-determined verbs that represent the Bloom’s stages indicating the level of course expertise. Questions are labelled as per Blooms taxonomy. Natural Language Processing is a Machine learning based method to classify the text, as it easily converts the text into tokenized format that can be used for classification. Classification models are developed using the Bidirectional Long Short-Term Memory networks and Convolutional Neural Network. Deep Learning toolbox of MATLAB 2023a is used to train and test the model. The CNN model has provided a validation accuracy of 85.71% for the labeled questions.
... Zhao and Li [6] developed an examination system using Java web technology, which combines client-side programming and server-side programming to provide functions such as question management, paper generation, and online testing. Yagci and Üünal [7] designed an adaptive online examination system that can automatically determine different sets of questions for each student through interactive means in order to more effectively assess their abilities. A framework for an online examination system based on blockchain was proposed by Sadayapillai and Kottursamy [8]. ...
Article
This paper mainly studied the automatic test paper generation and scoring problems in university English exams. Firstly, an automatic test paper generation model was established. Then, an improved genetic algorithm (IGA) was designed for intelligent test paper generation, and it was also used to automatically score answers to Chinese-to-English translation questions in terms of syntax and semantics. It was found that compared with the traditional GA and particle swarm optimization algorithm, the IGA method was faster in generating test papers, with an average generation time of 25 s, and had a higher success rate (94%), suggesting higher validity, and the difficulty and differentiation degrees of the test papers were closer to the preset values. The results of automatic scoring also had a correlation of more than 0.8 with the manual scoring results. The results prove the effectiveness of the automatic test paper generation and scoring method. It can be promoted and applied in practice to enhance the security and fairness of large-scale English exams, as well as achieve objectivity and consistency in scoring.
... In Designing and Implementing an Adaptive Online Examination System [5], a design and application of adaptive online exam system are carried out. Ability and skills of student in particular area of the knowledge plays an important role in case of adaptive exam system. ...
Article
Full-text available
Today there are two forms of training and education: Distance education and Conventional education. Mobile learning, or "M-Learning", offers modern ways to support learning process through mobile devices, such as handheld and tablet computers, smartphones and mobile phones. For increasing the learning awareness and understanding of teachers and students Mobile learning is necessary. Online examination is necessary part of the M-learning and secure authentication is the most important parameter for success of M-learning. Hence, conducting the exam in open environment and providing security to them where each student is using her or his mobile phones, laptops or tablets which are connected to the internet through Wi-Fi network. This paper discusses the implementation of secure exam management system. The paper discusses about system which consist of online examination, secure authentication and secure environment for examination.
Article
Full-text available
College are major center of higher education which ,by being affiliated with universities,play an important role in providing higher education.the quality of higher education today is a major demand for the development of a country .the assurance of which is guaranteed by grading by NAAC and all efforts are being made to upgrade the quality. In the present research paper examination a major part of education,has been analyzed as a hindering element in improving the quality of higher education.how the examination system is spoiling the education because any system is good to a limited extent but its too much becomes harmful. Due to this,the relationship between education and examination has been analyzed in this presente research paper and it has been found that how universities are contucting examinations and their adverse impact on providing education and the activities of higher education are an essential part of education.components are getting adversely affected by the examination itself.through this research paper ,the aim is to draw the attention of policy makers of higher education towards the problems arising from the current semester sstem and to provide suggestions for improving the examinatio.
Article
Full-text available
Article
In recent years, there has been an increase in online education. However, an adequate technique was not developed for online examinations. Some educational facilities have compiled assignments that students may copy and paste from the web, while others use remote proctoring, where a human proctor monitors what the students are doing online. Cheating in online tests remains common despite all the growth that has taken place in this field. This paper proposes an Artificial Intelligence (AI)-based system that can assist in the automatic detection of fraud in online tests. This method is both efficient and trustworthy. This system outperformed previous systems in experiments.
Article
Full-text available
Institutions of higher learning in developing countries are gradually embracing electronic examinations as a result of Covid-19 standard operating procedures that emphasize social distancing. However acquiring secure and acceptable electronic examinations has been a challenge as most of the available electronic examination systems are constrained by limited question types, highly reliant on keyboard typing, and lack of an electronic answer booklet. The electronic examinations framework with free handwriting presented in this paper seeks to address all the aforementioned challenges in-order to improve the usage and security of electronic examinations. To achieve the aforementioned goal, design science methodology was. Results obtained indicated that for usage and security to be eased, various parameters should be put in place i.e. user-friendly interfaces, authentication mechanisms, electronic examination setting, marking and attempt with electronic free-handwriting, electronic results, and a repository for student and lecturer data. The research output/artifact will be beneficial to Students, Lecturers and Institutional administrators.
Book
Ronald K. Hambleton; H. Swaminathan; H. Jane Rogers., The following values have no corresponding Zotero field: Label: B496 ID - 337
Article
Intelligence and adaptation have been very important factors in designing learning systems. While the research in the field has yielded much useful information, a lot more remains to be done. We suggest that adopting a granular structure reduces some of the complexity and consideration of the Teacher as an environmental context of the system is crucial for its success. The discussion is based on our experience of designing and implementing such a system and the feedback from a comparative independent study.
Article
Studies on the effects of computerized testing show that computer-based tests are not equivalent to their conventional counterparts, the results of item feedback are inconclusive, and computerized-adaptive tests can be as reliable as conventional tests with fewer items administered. The effects of the ordering of items, the ability estimation procedures, and the context of item administration in the estimation of parameters are some areas that require further investigation.
Web Tabanlı Sınav Sistemi. Unpublished master's thesis, Sakarya Üniversitesi, Fen Bilimleri Enstitüsü
  • O Akın
Akın, O. (2007). Web Tabanlı Sınav Sistemi. Unpublished master's thesis, Sakarya Üniversitesi, Fen Bilimleri Enstitüsü, Sakarya.
Uzman Sistemler, Atlas Yayıncılık
  • N Allahverdi
Allahverdi, N. (2002). Uzman Sistemler, Atlas Yayıncılık, İstanbul.
The evaluation of student performance via the fuzzy logic approach
  • H Karal
  • A Çebi
Karal, H. & Çebi, A. (2011). The evaluation of student performance via the fuzzy logic approach, IETC, Proceedings Book Vol 2, 1445-1451.
Educational research, methodology, and measurement: an international handbook
  • F M Lord
  • M L Stocking
Lord, F.M. & Stocking, M. L. (1988). Item response theory. In J.P. Keeves (Ed.), Educational research, methodology, and measurement: an international handbook, Pergamon Press, 269-272. Newyork.
Anti-phishing and pci security services
  • Netcraft Ltd
Netcraft Ltd. Anti-phishing and pci security services. Retrieved September 05, 2012, from http://news.netcraft.com.
Uzaktan eğitimde bilgisayar kullanımı ve uzman sistemler. The Turkish online journal of educational technology
  • H H Önder
Önder, H. H. (2003).Uzaktan eğitimde bilgisayar kullanımı ve uzman sistemler. The Turkish online journal of educational technology,2(3),142-6.
Research on the effects of administering tests via computers. Educational measurement: Issues and Practice
  • S L Wise
  • B S Plake
Wise, S. L. & Plake, B.S. (1989). Research on the effects of administering tests via computers. Educational measurement: Issues and Practice, 8(3), 5-10.