Content uploaded by Gabriele Frankl
Author content
All content in this area was uploaded by Gabriele Frankl
Content may be subject to copyright.
The “Secure Exam Environment” for Online Testing at the
Alpen-Adria-Universität Klagenfurt / Austria
Gabriele Frankl
e-learning Department
Alpen-AdriaUniversität Klagenfurt, Austria/Europe
gabriele.frankl@aau.at
Peter Schartner
System Security Group
Alpen-AdriaUniversität Klagenfurt, Austria/Europe
peter.schartner@aau.at
Gerald Zebedin
e-learning Department
Alpen-AdriaUniversität Klagenfurt, Austria/Europe
gerald@zebedin.at
Abstract: This paper aims at taking a glance at a particular new and flexible
solution of online-testing at the Alpen-Adria-Universität Klagenfurt. We present a very
recently implemented method that allows students to conduct their exams online and directly
on their own devices without the possibility to access their own files or non-specified
internet pages. A survey among the participating students in the pilot exams showed that the
general attitude of students is very positive. Especially among lecturers the demand for the
secure exam environment (SEE) for their courses inclined tremendously. For the future we
plan to further develop the environment, e.g. to offer an additional version for open book
exams or for specific software requirements.
Why Online-Testing?
The Alpen-Adria-Universität Klagenfurt (AAUK), located in the South of Austria/Europe, aims at
providing high quality teaching combined with excellent support of individual students. Although we are a
smaller university, offering appropriate support is becoming more and more challenging in times of increasing
numbers of students, especially in high-frequented study-programs. The AAUK counts 9,988 students which
are spread among the four faculties: Social Science and Economics, Engineering Science, Humanities and
Cultural Studies and the Interuniversity Institute for Interdisciplinary Research and Continuing Education. Out
of the whole range of study programs, Business Administration, Media and Communication studies as well as
Psychology and Educational studies became apparently the most popular programs.
Up-to-date teaching is an important and essential concern of the AAUK. However, contemporary
teaching requires indeed also up-to-date testing and examining methods. Exactly these issues shall be tackled
through the ideal implementation and usage of technologies. Therefore, the e-learning department of the AAUK
developed the “secure exam environment” (SEE) in cooperation with the System Security Group of the AAUK.
Seeking Efficiency
In order to guarantee and to meet high quality in teaching methods, the AAUK seeks to perfectly use
time resources wherever it makes sense. Processes that can be carried out automated, at least or much more
efficient than human beings, should not block time resources of lecturers. Instead, the gained time should be
used for processes requiring interpersonal communications or social dynamics. One area that perfectly fits the
requirements of a reasonable implementation of automation is testing. Actually, oral examination should be
preferred to a written exam. However, due to time restraints in many academic disciplines, multiple-choice
exams are conducted on a regular basis. Indeed, a smart implementation of multiple-choice questions can result
in appropriate testing instruments. However, the evaluation of the exams entails manual scanning or correcting
of the test scripts. Hence, the solution is online tests, which tremendously reduce the correction work.
Using Potentials
A potential, which immediately comes to one’s mind when thinking about online-testing, is the
automated evaluation of standardized questions, for example multiple-choice, cloze, numerical answers or
matching. However, there are many other features offered by online testing which are not so commonly thought
of. Firstly, for lecturers it is much more comfortable not to struggle with hand-written exams. Secondly,
questions can easily be created, stored, enlarged and organized. Over the time the question pool gets
comprehensive enough to reuse questions for future exams, fostering sustainability and decreasing work load.
Thirdly, Austrian students usually do not make use of their right to inspect the results of their exams by
personally contacting the lecturer and therefore lacking valuable feedback. This hurdle can be overcome with
the possibility of automated feedback in online exams. The lecturers have the possibility to generate automated
feedback for each question, and explaining the reason for the correct or incorrect answer. Additionally,
individual feedback can be provided. The students have the chance to view the feedback anytime, anywhere,
without the need of personal consultancy. Fourthly, the issue of cheating, which seems to be a common problem
in many European countries, is hindered. The lecturer can choose whether questions and/or answers are
randomly displayed or not. Thus, looking at the screens of colleagues is useless. In addition, since no other
materials are in use and both hands have to be used for typing, hidden and not allowed actions are to a large
extent eliminated. Finally, up-to-date teaching requires in many cases also up-to-date testing. For example,
some calculations are nowadays basically solved with software programs, e.g. Microsoft Excel. If there is no
computer available at the exam, these calculations have to be solved manually on the paper, which might be no
contemporary method anymore. As a matter of course this effects also the teaching. Especially this is not
desired if those manual methods do not have a learning effect at all.
Current Obstacles to Online-Exams
Online-exams are already in use since many years. In the majority of cases online-tests are carried out
in computer rooms, restricting online-exams to small groups. For larger groups of participants the majority of
computer rooms are too small to perform them online. However, the financial means to acquire mass-computer
rooms do not exist. Furthermore, universities are faced with the fact that technical devices become obsolete
after a few years. In order to overcome the barriers of computer rooms, a solution is needed to make use of the
anyway existing students’ technical resources. This offers the advantage that students are used to their own
devices, for example windows or apple-devices, and have the necessary software needed for particular courses
already pre-installed. Moreover, the effort to check devices in computer rooms before online-exams is omitted.
Certainly, when using the technical resources of the students, loan devices are offered in case of technical
problems.
Overcoming the Obstacles: The “Secure-Exam-Environment”
The SEE constitutes the answer to the before mentioned obstacles and problems that are currently
faced in academic daily business. By using the resources of the students, the developed application disables the
access to students’ own files and data as well as to internet addresses and links.
The AAUK is using the e-learning-platform “Moodle” (see http://moodle.com) as a teaching and
testing tool. Therefore, the usage of the SEE is embedded in Moodle.
Technical Architecture – Security by Design
Online applications and exams come both with massive security issues (see Weippl, E.R. 2005, Bartram &
Hambleton 2008, and Scheuermann & Björnsson 2009, pp. 84). Considering the electronic exam environment,
we have to consider three major aspects:
Security (for the Examiner)
Here, as with classical exams, cheating is the foremost problem. On one hand we have to prevent users
from accessing (electronic) materials like notes, books, local or online resources, which are not allowed during
the exam, and on the other hand we have to prevent impersonation.
It is easy to counteract impersonation by checking the photo on the student identity card at the
registration. During the exam it is also possible to compare the registration-data with the (number of) students
who started the exam in the Moodle course. Preventing users from accessing materials which are not allowed
during the exam is more complex. In contrast to other electronic exam environments (e.g. see SoftwareSecure
2011), we did not want to use special equipment. We wanted the students to use either their own or a loan
laptop. Hence accessing the Moodle server directly via a webbrowser running within the student’s OS is an
inefficient approach. In this case, dropping connections to Wikipedia or other online resources may be simple,
but cheating by the use of materials stored on the local hard drive is rather easy. Since we do not want to force
students to install additional software (like lockdown modules) on their personal laptops, we have to use our
own operating system in order to restrict the access to the local resources and programs (e.g. chat programs,
dictionaries or calculators in case they are prohibited during the exam. Since KNOPPIX
(http://www.knopper.net/knoppix/index-en.html) is (said to be) the Linux distribution with the broadest
hardware support, we decided to use a customized KNOPPIX version, which is provided as a live DVD or is
started via USB. In order to restrict the access to external (online) resources, we implemented corresponding
firewall rules. Unfortunately Moodle does not only provide the exam features, but also provides chatting
capabilities and (in most cases) course related material. So running an ordinary web browser in the KNOPPIX –
even when restricted with firewall rules – does not completely solve our cheating problem. Fortunately, the Safe
Exam Browser (SEB – see http://www.safeexambrowser.org/news_en.html for details), an open source project
supported by the University Giessen and the ETH Zurich, has been integrated into the Moodle-core 1.9.4. So by
restricting the access to the exam page only, cheating by means of the features provided by Moodle itself is not
a problem anymore. But the Safe Exam Browser comes with a little drawback again: at the time SEB is only
available for Windows XP, Windows 7 and MAC OS X. So we have to run a minimized WindowsXP which
provides exactly the necessary programs (currently the SEB only) in a virtual machine (VIRTUALBOX, see
http://ww.virtualbox.org) on the KNOPPIX system (see Figure 1 for a sketch of the system).
Summarizing, our secure exam environment comes with the following security features:
• Reduced KNOPPIX in order to support a very broad range of (private) laptops and prevent students
from accessing local resources and programs.
• KNOPPIX-Firewall in order to prevent users from accessing internet resources.
• Safe Exam Browser (integrated in Moodle) to prevent users from cheating by means of features
provided by Moodle.
In order to start an online exam by the use of the secure exam environment, users have to perform the
following steps
1. Booting KNOPPIX (from USB or DVD)
2. KNOPPIX automatically starts the WindowsXP-VM
3. WindowsXP automatically starts the Safe Exam Browser
4. The Safe Exam Browser automatically connects to the homepage of the AAU’s learning management
system Moodle.
5. Provide login and password for Moodle
6. Select the exam
Student’s Laptop
Uni-Klu Network
Customized KNOPPIX
KNOPPIX Firewall
VirtualBOX
Minimal WindowsXP
Safe Exam
Browser
Moodle Server
Internet
Local hard drives
Memory cards
External devices
Exam
Chat Materials
...
Google
Books
Wikipedia
...
Open exam
Other actions
Figure 1: The operating principle of SEE
Safety (for Examiner and Examinee)
Except lightning, classical exam environments (i.e. pencil and paper) do not depend on (information)
technology during the exam itself. In contrast to this, electronic exam environments massively depend on the
availability of (information) technology during the exam. At the time of writing, our system depends on the
online connection between the Safe Exam Browser and the Moodle Server. In case of a network failure, none of
the users can safe current results or proceed to the next question. So temporary storage of the complete exam at
the beginning of the exam and temporary storage of the answers (during network failures) is one of the
currently open problems.
Hardware failures of laptops are not a problem, because all answers provided up to the failure have
been stored on the server and the student can simply continue his exam on one of our load-laptops.
In order to provide a maximum availability of the network connection we only use cat5-wired LAN at
the time. WLAN is too error prone and additionally, one malicious user could perform a DoS attack on the
WLAN access quite easily and hence prevent all users from taking the exam.
In the context of exams, archiving the exams is another important aspect. According to the Austrian
Universitätsgesetz 2002 (see Republic of Austria 2011), documents related to written exams have to be archived
for at least six months (see Republic of Austria 2011, §79 (3)), whereas protocols of orals exams have to be
archived for at least one year (see Republic of Austria 2011, §79 (4)). Since we use Moodle for the exams, we
do not need to bother about archiving, Moodle does it for us automatically.
Privacy (of the Examinee)
The answers provided by the students and the achieved grades (called exam data in this paper) are
individual-related and hence according to the Austrian Data protection act (see Republic of Austria 2009) their
privacy has to be protected. We use encrypted and authenticated transmission lines between the SEB and the
Moodle server (HTTPS), and the Moodle server uses login/password-authentication for the students and the
lecturers and grants access to authorized persons only. Hence, we can guarantee the privacy of exam data.
Organizational Aspects
In order to successfully implement the SEE, organizational measures – before, during and after the
exam – have to be taken into consideration. This organizational embedding can be split into three stages.
However, the very first step is to inform the e-learning department of the university to ensure proper support
throughout the three phases.
Phase 1: Preparation Period
An essential stage is the careful preparation of online-exams. Lecturers offered support by the
e-learning department of the university to properly set up their online-exams. When preparing an online-exam
(creating online questions and composing the exam in Moodle), it is important to consider the security settings.
In order to make use of the extended security, offered through the combination of the SEE and the Safe Exam
Browser, the following setting has to be selected in Moodle: choose in the field “browser security” (under the
area “security” in the course settings) the option “require the use of Safe Exam Browser”. To guarantee that
only students, who are present in person can access the online-exam, enter the network address of the lecture
hall in the designated field (area “security”). Furthermore, it is advisable, that time and attempt constraints are
specified for the online-exam (e.g. fields “timing” and “attempts” under the course settings in Moodle). At this
point the lecturer can decide to automatically display feedback and/or results after the students have submitted
the online-exams.
Additionally, it has to be highlighted that the requirements for the SEE are electricity and network
access at each and every seat in the lecture hall. At the moment, there is a limited number of lecture halls that
offer these prerequisites also for larger groups of students. Therefore, it is highly important for the lecturer to
book the lecture hall early enough for the exam. Moreover, loan devices must be provided. On the one hand, it
cannot be assumed that all students have a mobile device. On the other hand, loan devices must be ready in case
of technical problems or breakdowns of the students’ own devices during the exam. The AAUK currently has
50 ACER TravelMate 8172 netbooks at its disposal, serving as loan devices for students.
For the exam itself, the students require a network cable and an USB flash drive, or a DVD for Apple
users, from which they boot the SEE. In order to guarantee the security of the application it is of utmost
importance that the USB flash drives or DVDs are returned after the exam. This action requires organization.
Therefore, the lecturer has to provide the e-learning department with a list of all students taking part in the exam.
The e-learning department adapts this list to their needs. The students have to – immediately before the exam –
confirm and underwrite the receipt of the devices. Further, the students have to be informed in due time before
the exam about the action chain for the online-exam. The e-learning department of the AAUK offers checklists
especially for students which are provided in the respective Moodle courses.
Phase 2: Execution Period
The implementation and execution of online exams requires a certain amount of time. Therefore, an
efficient and well-planned organization is vital. The optimal way is having several online-exams in a row. Loan
devices, network cables, USB flash drives, or DVDs are brought in the morning into the lecture hall and can be
used the whole day. Since the installation requires some assistants, the work load can be reduced tremendously
having several consecutive exams.
At the AAUK two test scenarios have already been successfully conducted. With smaller groups of
students, the available loan devices including network cables and power supply were prepared, installed and
safeguarded with Kensington locks before the students arrived at the lecture hall. Students were also allowed to
use their own devices if requested. This scenario was very efficient in the execution process, especially since
three exams were carried sequentially. For the students hardly any waiting time occurred, since they had only to
register and boot the device from the USB flash drive or DVD. Nevertheless, as the qualitative survey pinpoints,
this caused some disadvantages because some students were not accustomed to the loan devices and would have
had preferred to work on their own devices.
In the second scenario, the students used their own mobile devices (notebooks), with only a few
exception using a loan device due to non-availability or technical problems of the own device. In virtue of the
fact that nearly every device has for example a different key combination to boot, the installation period took
much longer which also increased support costs. However, the advantage was that the students were familiar
with the handling of their own devices, which is particularly beneficial in a stressful exam situation. The
disadvantage of this scenario will be compensated in the future by an information desk in the hall, where
students can try to boot a trial version of the SEE with their own devices. This includes both, the advantage for
the e-learning department to know in advance who needs a loan device and the advantage for students to get
familiar with the booting procedure. Hence, we can gain time in a real online-exam situation. Generally, the
tendency will be that over the time students become more familiar with the process, making online-exams with
the SEE a routine.
The actual process of an exam with the SEE is as follows: There are – depending on the number of
registered students- enough registration tables in the lecture hall and, as required, subdivided alphabetically.
This guarantees on the one hand a fast, structured and efficient registration of the students and on the other hand
it can be ensured that all loaned devices i.e. network cable, USB flash drive/DVD and if necessary a netbook
are returned completely. The loaning of the devices is confirmed by the students with their signature on the
respective registration lists at the registration tables. The loan devices are numbered so that any and every
participant is assigned to a seat, dissolving any friendship or support groups during the exam. The students take
a seat and make sure that the power and network cables are plugged in. Then they boot their device from the
USB flash drive or the DVD. The full instructions are projected in the lecture hall and e-tutors1 are on the
ground to support students during the starting process. After a successful booting process, students are
automatically directed to the Moodle site. There, they login with their user data and select the relevant course
with the specific exam.
If the lecturer changed the Moodle test settings (as highlighted under Phase 1), students will only be in
the position to start the exam once at a certain time and only have a certain amount of time to finish the exam.
Allowing only one attempt anticipates the possibility for students to submit their own exam and then log in
again and write the exam for a colleague. These settings additionally augment the security of online-exams. As
soon as the test is available, there is the possibility of real-time checking whether all students have started the
online-exam or not. As soon as the students have finished the exam they submit their results. After the editing
time of the test has expired the results will be submitted automatically. A supervisor checks again in real-time
whether all participants have properly submitted their exams. Then the students shut down their devices,
remove the USB flash drive (or beforehand the DVD) and return the USB flash drive/DVD, the network cable
and if applicable the loan device. The supervisors on the registration desks confirm the receipt on the
participation list. The returning of the devices is a prerequisite for receiving a grading of the course.
Phase 3: Post-Processing
Immediately after the online-exams, the safeguarded devices are removed by the e-learning department.
The lecturer can in this phase decide when and in which form the results of the online-exam are displayed.
Automated feedback, already created with standardized questions can now be displayed. The advantage is that
students can immediately receive feedback and see e.g. the correct answers and an explanation to the questions.
In Moodle there are many ways how the exam results can be released by the lecturer. For example, students are
only allowed to see how many points they have achieved, which answer they have written or with standardized
question types, which answer would have been correct. Additionally, there is the possibility for students to give
feedback (anonymous if applicable) on the online-exam in the Moodle course.
The “Secure Exam Environment” in Practice
Until now the SEE has been successfully implement in pilot test as well as in real exam settings. The
demand of lecturers to use this way of online-exams is extremely high. This is on the one hand due to the
success of the pilot tests but on the other hand also a result of the extremely high pressure lecturers are exposed
to, regarding to the evaluation of exams, especially in the high frequent study programs.
Current Experiences
In June 2011, as a pilot test five final exams have been conducted by means of online-exams. The
courses come from the area of business administration, geography, media and communication science as well as
from educational studies. In total 136 students participated in the online-exams. For coming autumn, additional
online-exams are planned. In the winter term 2011/12 online-exams will become widespread in the testing
culture of the AAUK.
1E-tutors are the support personnel from the e-learning department
Qualitative Survey
Students, who used the SEE for their final exams, were asked to give feedback after the online-exam in
their respective Moodle course. Four out of the five courses participated in the survey. Out of these four courses
a total of 109 students participated in the online-exam. 55 out of 109 (50.5 %) submitted their feedback. This
high response rate is attributable to the fact that students of two courses were awarded with points for filling out
the feedback. These points accounted to the overall course evaluation (5 points for the feedback – 155 possible
points in total). All four exams were composed of standardized as well as open questions. For the two exams
out of the discipline of geography, standardized questions dominated the exam (participating students: 44,
feedback received from 8 students). For the exams from the disciplines of media and communication science as
well as educational science, open questions prevailed in the exam (participating students: 65, feedback received
from 47 – with 5 points for the feedback). The majority of the students that participated in one geography exam,
both exams of the media and communications studies as well as educational studies used loan devices. The
participants of the business administration and the second geography exam could decide freely whether to use
loan devices or their own devices.
The questionnaire
The feedback itself compromised five questions:
1) How do you assess the online-exam in general? (standardized answer possibilities: very
negative/negative/positive/very positive)
2) In case you have used your own device for the online-exam and technical problems occurred: Which
problems happened and which type/brand of device did you use? (Open question. This question was
important for the further development of the SEE which shall not be handled in this paper).
3) Which advantages and disadvantages do you associate with online-exams (vs. traditional paper exams)?
(For example, did you need more/less time for the online-exam compared to paper exams?) (Open
question)
4) Would you like other courses to implement online-exams? (standardized answer possibility: yes/no)
5) Additional comments on online-exams. (open question)
Results of the survey
In general, students evaluated the online-exam as positive. Question 1 was directed towards the general
impression of online-exams which nobody of the students considered as “very negative” (0 %). Nine
respondents (16.4 %) answered with “negative”, 35 respondents (63.3 %) with “positive” and eleven
respondents with “very positive” (20 %) (cf. to figure 2).
Figure 2: General perception of online-exams
The dominating advantage, from the students viewpoint, (questions 3 and 5) is that typing on the
0" 5" 10" 15" 20" 25" 30" 35"
very"negative"
negative"
positive"
very"positive"
mobile device saves a tremendous amount of time compared to the classical paper exams (25 entries). This
gained time can be used more efficiently for answering questions. With 16 entries, the second most cited
advantage is the very quick and unproblematic correction of the questions by the students during the exam,
which avoids messing up the answer. Seven respondents additionally indicated that when revising the answers
the overview is maintained and the answers are proper and structured. The prompt evaluation of standardized
questions has also been positively mentioned (10 times). Seven entries indicated that online-exams are a
modern and up-to-date form of testing and that nowadays everybody should be familiar with the handling of
computers. One entry explicitly highlighted that online-exams are a particular advantage for courses which
require computers. Seven students were pleased with relieving the hand, since manual writing often causes pain.
Six students indicated that their hand-written answer are sometimes very chaotic and unreadable and that they
are concerned whether the lecturer is in the position to decipher their results or not. Another six respondents
regarded the improved readability of the typed language as an advantage and relief for the lecturers. Five
answers dealt with the much faster (and once also mentioned easier) handling of the whole exam.
Additional advantages from the position of students are less waste of paper and saved printing costs (4
entries). A further big advantage mentioned by the students is that the problematic of cheating is tremendously
reduced, inter alia by random display of questions and mixed answers (3 entries). Further advantages (one entry
each) are that there can be no mixing–up of the exams when submitting them; online-exams are very fair due to
the fact that everybody is equally restricted to the same time span and that it is fantastic to have a look at the
whole, by the lecturer corrected, exam which improves transparency of grading.
One student said, “I personally appreciated the fixed seating and I had the feeling that I could
concentrate very well on the exam”. Another student said, “I experienced this type of exam very comfortable
even though time was short. Writing the exam on the computer did not give me the feeling of an exam that’s
why I was able to work much more relaxed! This is one of the most important advantages of online-exams”.
The most frequently mentioned disadvantage- from a student perspective -is the additional workload
for lecturers and supporting staff for online-exams (11 entries). Ten respondents highlighted the unfamiliar
handling of the loan devices. However, two students out of ten relativized this by mentioning that they
familiarized very fast. Nine students estimated that they need more time online compared to traditional paper
exams. Two of them attributed this to the fact that it was their first online-exam. Three thought that they are not
very fast with typing and one person mentioned being faster when writing by hand. Four respondents pinpointed
that the overview of the exam respectively the checking of the answers is more difficult online. Another four
students mentioned that they feared their answers will disappear or that they forget to save their answers.
Furthermore, four participants said that the waiting time at the beginning – until everybody is ready - is
annoying and the increased typing mistakes due to time pressure were another issue of concern.Three answers
dealt with the fact that it is more difficult to collect one’s thought online. Others (one entry each) mentioned
that reflection time before writing is shorter and that paper exams are better arranged.The facts that paper exams
are easier to carry out, online-exams are very exhausting for the eyes and multiple-choice questions do not
guarantee learning success were mentioned each time once. One student was concerned about the consequences
in case of handing back a broken USB flash-drive. Another person mentioned that the noise of 51 typing
students at the online-exam in one room is loud and can be distracting.
In some points the feedback of the students is very contradictory, e.g. concerning the writing/typing
speed. Four students shared the opinion that the automatic countdown (time display) is an excellent orientation
during the exam; two students thought that the countdown could be a problem for other students and six
students felt under pressure due to the time display. As previously mentioned, ten students had problems with
the handling of the loan devices whereas two students where very pleased about them. Another interesting fact
is that five students thought that for other students – not typing so fast – online-exams could be a problem.
Additional four students said that they did not encounter technical problems but there could be some. Five
studentsindicatedthat they need as long for online-exams as for traditional paper exams.
As an intermediary result it can be mentioned that for online-exams a gap between media-competent
students and those who are not originates (although the latter group is decreasing continuously). However, a
slight digital divide can be recognized at this point.
The current mood towards online-exams can also be derived from questions three and five. In total, the
online-exams have been called 35 times “awesome”, “exciting”, “fascinating”, “very interesting”, “very good”
or “very positive”. Students indicated that they are “thrilled” or “had fun”. 20 students explicitly wrote that they
would participate again at any time in an online-exam and they would appreciate if online-exams are integrated
in the university daily business. Another two students said that they wish further development of the
online-exam but using only their own devices.
19 students gave feedback on the online-exam and its processing. It was highlighted that the exam was
perfectly organized and/or supported. Additionally four students where surprised not encounter any technical
problems, thought it was a pilot test. Six students are looking for the further development of standardized
questions.
The specific question, whether students would like to have online-exams also for other courses
(question4) showed the following picture: 37 students (67.3 %) agreed whereas 19 students (32.7 %) disagreed
with the question (cf. to figure 3).
Figure 3: Online-exams for other courses
Limitations, open problems and future work
One of the current restrictions of online-exams is the required network connection. Wireless LAN is
still prone to failure, especially for larger groups of students. However, for the future it is planned to provide a
wireless LAN access point. Out of this restriction another obstacle evolves, namely that lecture halls require
LAN and power sockets at each seat. Unfortunately, not all lecture halls fulfill these requirements and
retrofitting is extremely expensive. The obstacle with the LAN sockets could be overcome with access points.
However, using the netbook only with battery for exams is too risky. Additionally, another obstacle is the
limited number of loan devices and sometimes the occurring technical problems with students’ devices.
After testing the prototypical implementation of SEE, we have identified the following points for
future work:
• WLAN connection of the employed laptops.
• Booting the Secure Exam Environment via Pre Boot Execution (PXE)
• Temporary storage of exam and provided answers in order to handle transient (W)LAN and connection
and power losses.
• Configuration support and usage of profiles, for exams where additional programs (e.g. calculator,
spreadsheet, paint) or external resources (e.g. dictionaries) are allowed.
• Support of tablets like iPad or Android devices.
Outlook and implications
Starting with autumn 2011, the implementation of the SEE will drastically augment. Hence, we have
the opportunity to receive extensive and informative feedback for further improvements and developments of
the application. The feedback received up to now results from pilot tests that were based on different exam
scenarios. In the future the different questions and test processes require much more differentiated investigation.
As already mentioned, in order to improve the SEE, it is planned to set up an information desk at the university
hall where students can trial test the booting from the USB flash drive or from the DVD with their own device
0" 10" 20" 30" 40"
no"
yes"
before the exam time starts.
Additionally it is planned to develop a version of the SEE for open book tests. Until now, voluminous
scripts have been printed out for exams which is simply not up-to-date anymore. In the future, it should become
possible to use electronic documents.We are convinced that due to the augmented flexibility, efficiency and
optimization of learning and teaching that can be achieved through new possibilities in the field of e-learning,
teacher and students will receive optimal support in their individual learning/teaching habits and methods. The
developed “secure exam environment” is one possibility of many being offered to students and lecturers.
We are just at the beginning of a very exciting new period of teaching and learning methods that enrich
current and classical concepts. One aim is to change the existing exam culture. In many disciplines traditional
methods for performance assessment are in terms of didactics completely inappropriate. However, the classical
testing is in many cases much better than its reputation. Sometimes it is necessary to study hard facts and to test
these. Exactly for such situation, classical knowledge appraisals are very adequate. However, it is necessary to
communicate lecturers how good exams are composed. A guideline to create useful and high quality multiple
choice questions was developed by August Fenk from the AAUK. The guideline will be available for lecturers
from beginning of September 2011.
If classical exam methods are handled and supported electronically, time is saved for those areas of
teaching that require face-to-face a lot of time and that currently miss out. In particular these are in-depth
discussions, reflections, oral exams, if applicable, or the coaching of these areas. This also contributes to the
aim of the AAUK re-academization process of teaching. Nevertheless, we have to clearly point out that
face-to-face teaching at the AAUK is still very important and should remain as significant as it is. The AAUK is
trying to develop alternative and the best possible forms of performance assessment and testing of knowledge,
since, as one student mentioned in the feedback: "If you take the students' fear of the questions, then tests are,
being traditional or online, a good thing."
References
Bartram D. & Hambleton, R. K. (2008). Computer-Based Testing and the Internet: Issues and Advances, John
Wiley & Sons.
Republic of Austria (2009). Datenschutzgesetz 2000 (DSG2000), BGBl. I Nr. 165/1999, Online (last visit
26.8.2011)
http://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10001597
http://www.dataprotection.eu/pmwiki/pmwiki.php?n=Main.AT
Republic of Austria (2011): Universitätsgesetz 2000, BGBl. I Nr. 120/2002, Online (last visit 26.8.2011)
http://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=20002128
Scheuermann, F. & Julius Björnsson, J. (2009). The Transition to Computer-Based Assessment, report available
Online (last visit 26.8.2011) http://www.gesci.org/assets/files/reporttransition.pdf
SoftwareSecure (2011). Securexam. Online (last visit 26.8.2011), http://www.softwaresecure.com
Weippl, E.R. (2005). Security in E-Learning (Advances in Information Security), Springer.
