Content uploaded by Jeremy B Williams
Author content
All content in this area was uploaded by Jeremy B Williams on Feb 01, 2018
Content may be subject to copyright.
The efficacy of final examinations: A comparative study
of closed-book, invigilated exams and open-book,
open-web exams
Jeremy B. Williams and Amy Wong
Jeremy B. Williams is Chief Academic Officer at Knowledge Universe Education in Singapore and an
adjunct professor in the School of Economics and Finance at Queensland University of Technology,
Australia. Amy Wong is an assistant professor and Associate Dean of Faculty Affairs, U21Global,
Singapore. Address for correspondence: Professor Jeremy B. Williams, 15A Changi Business Park
Central 1, #01-01 The Eightrium, Singapore 486035. Email: jeremywilliams@kueducation.com
Abstract
Educators have long debated the usefulness (or otherwise) of final examina-
tions; a debate that has typically revolved around the relative merits of closed-
book exams, open-book exams, take-home exams or their substitution by some
other assessment format (eg, project work). This paper adds a new dimension
to the debate by considering how the final examination assessment instrument
might be enhanced through harnessing the power of technology, more specifi-
cally, how the learner experience of the final examination might be made more
authentic and, in the process, more constructively aligned with stated learning
outcomes. The authors report on the latest findings of an ongoing research
project evaluating the effectiveness of ‘open-book, open-web’ (OBOW) exami-
nations delivered by an online university, vis-à-vis a closed-book, invigilated
alternative. Earlier research had indicated that the OBOW model receives the
strong endorsement of students in a number of respects, most particularly the
quality of the learning outcomes.
Introduction
After very little change in the university sector for several centuries, there has been
something of a ‘mini-revolution’ in the last quarter of a century or so. The learner
profile has changed socially, culturally and economically as new universities have
emerged, enrolling an increasingly diverse population of students, and—along with
this ‘massification’ of higher education (Carrier, 1990)—there has been spectacular
change in the area of educational technology. These changes have been reflected in
professional journals and conferences dedicated to teaching and learning, which have
produced a steady stream of literature reporting on a wide range of research projects
including, among other things, experiments with new classroom techniques, modes of
British Journal of Educational Technology Vol 40 No 2 2009 227–236
doi:10.1111/j.1467-8535.2008.00929.x
© 2009 The Authors.Jour nal compilation© 2009 Becta. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ,
UK and 350 Main Street, Malden, MA 02148, USA.
flexible delivery, online pedagogies and the increasing use of multimedia (see, eg,
Sims, 2004). Yet, despite the quite momentous transformation that has taken place,
one aspect of university life has barely changed at all, namely the end-of-semester
examination.
Indeed, the final examination is a university institution that would appear to be off-
limits as far as innovation is concerned. To put this into context, while faculty and
students alike will not stray too far from a computer as they go about their daily
business, it is still the norm for examinations to be conducted using pen and paper. Does
this imply, therefore, that some element of modern learning theory might be sacrificed
if it were abandoned in favour of some alternative instrument? Or, given it is still the
most commonly administered summative assessment instrument in universities today,
is there some other special intrinsic value attached to a closed-book, invigilated exam
that justifies its continued use?
These questions lie at the heart of this paper, which reports on an ongoing research
project comparing the relative merits of a closed-book, invigilated examination and an
alternative model referred to as an ‘open-book, open-web’ (OBOW) exam. The rationale
for such a research project arose out of a genuine desire to understand the preference
for a traditional examination format when no academic literature appears to exist that
extols the virtues of such an assessment instrument or provides a defence for its con-
tinued use. In the absence of such a literature, it made sense, therefore, to collect and
analyse data relating to the student experience with closed-book, invigilated examina-
tions and the OBOW alternative. A number of dimensions were identified for the pur-
poses of comparison, but of particular interest to the authors of this paper were the
relative opportunities for cheating and the depth of learning. The reason for this was
that these two factors are those most frequently cited—anecdotally at least—as the
justification for the retention of the closed-book, invigilated exam over the open-book,
non-invigilated alternative. Or, as it might be summed up in the departmental coffee
room vernacular, there is no substitute for the supervised pen and paper exam because
‘it’s the only way we can be sure that they know their stuff’.
In the sections that follow, there will first be a discussion of the institutional context of
the research project, the events that led up to the introduction of the OBOW exam
format and the defining characteristics of this instrument. There is then coverage of the
methodology employed in the conduct of the study, the results that emerged from it and
the inferences that might be made following analysis of these results.The authors of the
paper conclude that while neither instrument offers the perfect solution, the best alter-
native is to opt for the assessment instrument that delivers the superior learning out-
comes. In this instance, the evidence provides a strong endorsement for the OBOW
examination format.
Context
U21Global is a completely online university servicing approximately 4000 students in
around 60 countries. In operation since July 2003, it is backed by Universitas 21 (U21),
228 British Journal of Educational Technology Vol 40 No 2 2009
© 2009 The Authors. Journal compilation © 2009 Becta.
a network of 21 research-oriented universities spanning four continents. The U21
universities set the quality assurance benchmarks for all aspects of course delivery,
including appointment of faculty, development of content and monitoring of assess-
ment standards. One of the hallmarks of the U21Global operation has been the
firm commitment to a constructivist pedagogy where the learner takes centre stage
(Bostock, 1998; Huang, 2002). Following Biggs (1999), there is ‘constructive align-
ment’ of pedagogy, curriculum and assessment in that a case-based, problem-solving
approach is integral to an authentic assessment regime. This was not always the case,
however, as when classes first commenced, a closed-book, invigilated examination was
a mandatory element of all courses. This changed in May 2004, when it became clear
through feedback from faculty and students alike that this examination model was
failing as a summative assessment instrument because it was not consonant with the
constructivist pedagogy pervading U21Global courses and the attendant case-based,
formative assessment regime. The OBOW format was devised by drawing on earlier
experience that one of the researchers had with a similar model at a campus-based
business school at a university in Australia. Here, the idea had met with considerable
resistance and it ultimately failed to receive the endorsement of the school teaching and
learning committee as a legitimate summative assessment instrument. Conservative
forces also prevailed at U21Global in early 2004, and it was only after a lengthy evalu-
ation period that the OBOW model was officially sanctioned as the modus operandi for
final exams (see Lam, Williams & Chua, 2007; Williams, 2006).
Characteristics of OBOW
The defining characteristic of the OBOW approach is a commitment to authentic
assessment (see, eg, Herrington & Herrington, 1998; Wiggins, 1998). It fosters under-
standing of learning processes in terms of real-life performance as opposed to a display
of inert knowledge, and learners are presented with unstructured problems that
require the application of relevant skills and knowledge, rather than selection from
predetermined options as is the case with multiple-choice tests, for example. Most
importantly, it is a model that emphasises the importance of learner-directed discovery
of knowledge, or what Dalgarno (2001) refers to as ‘endogenous constructivism’,
which begins with the learner’s internal cognition and shapes their understanding
of the external environment through that mental lens. Such an approach engages
students, which, in turn, induces deeper learning.
While each OBOW examination is unique, one common feature is that the learner is
placed in the role of decision maker (eg, auditor, consultant or advisor). This is a critical
element of the OBOW model, as role play provides an effective bridge between a learn-
er’s education and the social context of their professional practice (Vygotsky, 1978); the
role of ‘expert witness’ serving as a useful mechanism for the validation of a student’s
learning in their own mind. Another critical element is that at the heart of every OBOW
exam is a contemporary real-world problem brought to life through the use of hyper-
links to websites and streaming media that serve to enhance the authenticity of the
problem.
Efficacy of final examinations 229
© 2009 The Authors. Journal compilation © 2009 Becta.
The OBOW model uses a very simple template that incorporates the design principles
outlined above (see Williams, 2004). The main objective is to produce a mini-case (‘The
Context’) that gets students to think conceptually about a problem, applying the skills
and techniques they have acquired in their study of the course in question. Having set
this context, the definition of the assessment task (‘The Task’) might amount to no more
than a paragraph or two that describes the role the learner must act out. The ‘Guide to
the Task’ that follows the assessment task definition is not to ‘spoon-feed’ the student
but to ensure that the task is not so unstructured that the student is either struck by
‘writers’ block’ or they fundamentally misinterpret the problem.
The final section of the template gives very specific instructions about preparation and
submission of the student’s response that deter unethical practice on the part of the
student. The work must be submitted electronically to permit use of plagiarism detec-
tion software but, more importantly, the student’s answer must make direct references
to course-specific materials. This means that an accomplice would first have to become
familiar with the subject materials, made more difficult given the time period allowed to
complete the task is sufficiently tight. Meanwhile, buying an assignment from an online
‘cheat site’ is not an option given the task is unique and highly contextualised.
In summary, the OBOW model represents a serious attempt to engage students rather
than alienate them. The opportunity for academically dishonest practice is less because
of the way these examinations are structured, but so is the temptation to resort to this
kind of behaviour in the first place. Students readily relate to the task that lies before
them as they can see the point of it. By contrast, the closed-book, invigilated exam
encourages a strategy of ‘cramming’ the night before and ‘data dumping’ on the day,
with little knowledge retention thereafter. The OBOW exam, meanwhile, is thoroughly
grounded in an authentic context, and learners have an opportunity to apply their
newly constructed knowledge in a meaningful way.
A position frequently adopted by those defending the continued use of closed-book,
invigilated final examinations is to state that students will cheat unless they are super-
vised. This assumes (1) that cheating is an easy thing to do within the OBOW model,
and (2) students do not cheat in invigilated examinations. Both assumptions are chal-
lengeable and have been tested in earlier research (Williams, 2006) and the current
project reported here.
Methodology
Earlier research conducted 3 months after the launch of the OBOW format surveyed all
students who had completed both formats of examinations offered by U21Global. This
preliminary study made use of a 10-question online questionnaire that focused on the
relative merits of each. Broadly speaking, the questions focused on the relative depth of
learning, real-world relevance, the congruence of the examination instrument with
U21Global pedagogy, the time allowed for the examinations, overall preferences regard-
ing examination format and the opportunities for plagiarism and cheating (Williams,
2006). In this study (n=54), all students either agreed (27%) or strongly agreed (73%)
230 British Journal of Educational Technology Vol 40 No 2 2009
© 2009 The Authors. Journal compilation © 2009 Becta.
that, overall, OBOW examinations were preferable to a closed-book, invigilated exami-
nation format; the remaining options on the 5-point Likert scale—strongly disagree,
disagree and neither agree nor disagree—receiving no votes. Indeed, all of the results were
similarly resounding in their endorsement of the OBOW model. On the issue of oppor-
tunities for plagiarism and cheating, many students took a neutral stance. Overall,
results indicated that it might be slightly more difficult to cheat in the OBOW situation,
but this was not deemed significant, there being roughly equal opportunity in the view
of the students.
In the latest study (June 2007), the same dimensions were explored as in the pilot study,
but with an improved methodology and a larger sample size. The main difference was
that the same questions were posed in relation to each exam instrument with a view to
conducting a gap analysis of the average scores to ascertain the relative importance of
each dimension. (In the earlier study, respondents were presented with a single set of
questions, each question including reference to both OBOW and closed-book exams.) In
addition, qualitative data was elicited, where students were asked to provide further
comments with regard to each question (see Appendix 2). Data were collected over a
period of 3 weeks and participants were students from U21Global who had taken at
least two subjects since the commencement of their programme. The participants were
recruited via an email message inviting them to take part in an online survey that
would take approximately 10 minutes to complete. In addition to the actual email, two
reminders were sent at 1-week intervals. The survey comprised 18 items; nine items
were solicited with regard to a closed-book, invigilated exam model, while the other
nine items applied to the OBOW exam model (see Appendix 1). A total of 91 (13.1%)
students responded to the email invite. Eighty-five per cent of the respondents were
male, most of them (59.1%) were between 30 and 39 years old, 30.7% of them had
completed two to five subjects at U21Global, while 29.5% of them completed more than
13 subjects. The profiles of the respondents were similar to the actual profiles of stu-
dents at U21Global.
Results
The results of the 2007 study reflect the findings of the 2004 study in that—in the
judgement of the students—OBOW is superior to the closed-book, invigilated option on
all counts. On the issue of opportunities for cheating, the result is also the same with
both the quantitative and qualitative data inferring that there are broadly similar
opportunities. One respondent was of the view that ‘It’s hard for students to cheat in a
well-invigilated exam’, but another commented, ‘The main disadvantage of a closed-
book exam is cheating by students’. Meanwhile, open comment in relation to OBOW
indicated that students felt that the OBOW exam is designed in such a way that cheating
is difficult given the exam is based on a current case study that was developed and
customised, taking into consideration the theories, concepts and issues covered in the
discussion boards and assignments. As one student put it, ‘Exams are designed in such
a way that cheating cannot take place’. Further, students are aware that the online
nature of the exam meant that plagiarism, fabrication and cheating can be detected
Efficacy of final examinations 231
© 2009 The Authors. Journal compilation © 2009 Becta.
more easily as compared to a closed-book, invigilated exam. ‘There is a real problem’,
stated one respondent, ‘if the examination performance is out of keeping with other
assignment performances’. This sentiment is endorsed by another student who wrote, ‘I
do not think an OBOW exam makes it any easier for a student to cheat, as it is very easy
now to pick up on material that has been copied and pasted from websites and profes-
sors would [also] be able to tell if the style of the exam answer was consistent with that
of previous material [written] throughout the module’. However, no system is perfect
because as one student noted, ‘[A] third party can be engaged to do the assignments,
discussion boards and exams’.
Overall, however, our findings are counter to a commonly held view that information
technology provides new situational opportunities for dishonest behaviour (eg,
McMurtry, 2001) and that, as a result, cheating should be made easier, faster and more
convenient, as students can share exam information via chat rooms, plagiarise from
the Internet or share exam questions via email between classes (Kleiner & Lord, 1999).
The findings are also inconsistent with a study conducted by Chapman, Davis, Toy and
Wright (2004) which reported that a relatively high percentage of students have
already been involved in academic dishonesty in a web-based testing situation. It could
be, of course, that the design of the OBOW instrument is the key determinant, a factor
possibly given less attention in the Chapman et al study. In our study, even though the
opportunity for cheating in OBOW was ranked slightly higher on this occasion (the
reverse being true in the pilot study), the difference is small and, at 0.2, the smallest
difference registered of all the dimensions being considered (see table 1).
While cheating does not emerge as an important indicator, the factor considered
most important in favour of OBOW was flexibility regarding the location of the exam
(+1.96), as students appreciate the fact that, with an OBOW, they can easily manage
their exam schedule to fit better with work and family arrangements. ‘[I] couldn’t really
take the course otherwise’, stated one student. Furthermore, they do not have to rush to
Table 1: Closed-book, invigilated exams versus open-book, open-web (OBOW) exams: a gap analysis
Dimension of examination instrument
Closed-book
(a)
OBOW
(b)
Gap
(b–a)
Time allowed for exam (3 hours vs. 24 hours) 3.49 4.19 0.7
Location (specific time/place vs. flexible) 2.63 4.59 1.96
Aligned with pedagogy 3.58 4.34 0.76
Quality learning outcomes 3.15 4.30 1.15
Intellectually challenging 3.51 4.49 0.98
Suited student’s learning style 3.06 4.43 1.37
Format relevant to business/professional education 2.95 4.42 1.47
Exam content engaging 3.38 4.41 1.03
Structure meant students could cheat 2.63 2.83 0.2
Note: 5-point Likert scale scoring system used: 5 =strongly agree,4=agree,3=neutral,2=dis-
agree,1=strongly disagree.
232 British Journal of Educational Technology Vol 40 No 2 2009
© 2009 The Authors. Journal compilation © 2009 Becta.
an unfamiliar place to take the exam. Instead, they can complete the paper in the
comfort of their own home/office, which serves to reduce their stress levels, and
improve their prospects of a good exam performance. The second most important factor
is that the OBOW is in a format relevant to business/professional education (+1.47).
In the business world, students are normally given a certain time frame to conduct
research and analysis as well as develop relevant strategies. Rarely, if ever, are they
required to go into a room, not talk to anyone and solve a work problem with their
computers turned off. Moreover, an OBOW is business focused in terms of context and
format, as exemplified by the following comment: ‘I think the exam format has also
helped me to learn more about business writing as the questions have all been very
practical and are about putting forward an argument as an expert in the area that
you’ve studied. As they are very practical, the questions enable you to think of many
different answers and responses which make the exam more relevant to real life busi-
ness situations’.
Other important factors in favour of OBOW include their being suited to students’
learning styles (+1.37). As one student put it, ‘I work better if I can reflect on my
thoughts and go back to something a few times to perfect it. In a 3-hour exam, I am
racing against the clock to try and get as much down on paper as possible, rather than
really focusing on what the question is asking’. This sentiment is consistent with the
superior quality of learning outcomes (+1.15), reinforced by students’ comments such
as the following: ‘You have the opportunity to do a quality, referenced piece of work
... more time can be spent on analysis and thinking, thereby entrenching learning as
opposed to regurgitating from memory and being time pressed; I felt more engaged in
this model of learning, ... the amount of analyses done and discoveries made through
the use of the Internet during the exam helped increase my knowledge and skills’.
Finally, the engaging content (+1.03) and the intellectual challenge of OBOW (+0.98)
emerged as important factors. ‘[I] got into trouble with [my] partner for saying that one
was fun!’ stated one student. Meanwhile, another commented that ‘All of the questions
I’ve had for the OBOW exams have been challenging and have made me sit down and
consider the question for at least an hour before I’ve actually started writing. I wouldn’t
have been able to allow this much time in a 3-hour exam, which I think would have
made my answer less considered and less accurate’. In addition, most students agree
that ‘case studies used in the OBOW exam allow for deep analysis, and access to Internet
research also enhances quality of learning and exam submission’.
Considered as a whole, the findings would seem to highlight the importance of authen-
tic assessment principles, such as the incorporation of real-life learning process and
unstructured problems that require the application of relevant skills and knowledge.
This further engages students and inculcates deeper and enriched learning.
Summary and conclusion
University examinations continue to be dominated by closed-book, invigilated pen and
paper tests. It is argued here that this is something of an anachronism given the human
Efficacy of final examinations 233
© 2009 The Authors. Journal compilation © 2009 Becta.
capital needs of a knowledge economy, not just because of the absence of technology
that is used routinely in everyday business and commerce, but because this type of
examination instrument is incompatible with constructivist learning theory that facili-
tates deep learning. It is further argued that a commitment to authentic assessment will
pave the way for a different type of final examination, where real-world problems are
allowed to take centre stage, and multimedia can be harnessed to provide the learner
with a more engaging experience. With greater engagement, this, in turn, can yield
better results in terms of the depth of student learning.
An ongoing research project with an open-book, open-web examination format at
U21Global has yielded positive results in terms of student perceptions of this examina-
tion instrument compared with the invigilated, closed-book type exam. Importantly,
OBOW is a transferable model that can just as easily be administered in an on-campus
setting as online, and while there will always be a small number of students who will
cheat, the main priority should be to focus on the higher quality learning outcomes of
the majority, rather than set up an entire system to stop a small minority. Certainly, if
there is roughly equal scope for cheating (as the results of two student surveys would
seem to suggest), then it would make sense to opt for the model that maximises student
learning.
A limitation of this study involves the methodological considerations pertaining to
measurement issues, sampling units and the number of variables investigated. Subse-
quently, this limits the generalisability of the findings reported. Hence, replications with
more representative samples of respondents and extensions of measurement scales are
planned for the future.
References
Biggs, J. (1999). Teaching for quality learning at university. Oxford: Oxford University Press.
Bostock, S. J. (1998). Constructivism in mass higher education: a case study. British Journal of
Educational Technology,29, 3, 225–240.
Carrier, D. (1990). Legislation as a stimulus to innovation. Higher Education Management,2,1,
88–98.
Chapman, K. J., Davis, R., Toy, D. & Wright, L. (2004). Academic integrity in the Business School
environment: I’ll get by with a little help from my friends. Journal of Marketing Education,26,3,
236–250.
Dalgarno, B. (2001). Interpretations of constructivism and consequences for computer assisted
learning. British Journal of Educational Technology,32, 2, 183–194.
Herrington, J. & Herrington, A. (1998). Authentic assessment and multimedia: how university
students respond to a model of authentic assessment. Higher Education Research and Develop-
ment,17, 3, 305–322.
Huang, H. M. (2002). Toward constructivism for adult learners in online learning environments.
British Journal of Educational Technology,33, 1, 27–37.
Kleiner, C. & Lord, M. (1999). The cheating game. U.S. News & World Report,127, 20,
54–63.
Lam, W., Williams, J. B. & Chua, A.Y. K. (2007). E-xams: harnessing the power of ICTs to enhance
authenticity. Educational Technology & Society,10, 3, 209–221.
McMurtry, K. (2001). E-cheating: combating a 21st century challenge. Teaching and Higher
Education Journal,29, 4, 36–41.
234 British Journal of Educational Technology Vol 40 No 2 2009
© 2009 The Authors. Journal compilation © 2009 Becta.
Sims, R. (2004). Trends, fads and futures: 21 years of educational multimedia: where have we got to,
and where should we go? Proceedings of World Conference on Educational Multimedia, Hypermedia
and Telecommunications 2004 (pp. 3755–3766). Chesapeake, VA: AACE.
Vygotsky, L. S. (1978). Mind in society: the development of higher psychological processes. Cambridge,
MA: Harvard University Press.
Wiggins, G. (1998). Educative assessment. San Francisco, CA: Jossey Bass.
Williams, J. B. (2004). Creating authentic assessments: a method for the authoring of open
book open web examinations. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds),
Beyond the comfort zone: proceedings of the 21st ASCILITE Conference (pp. 934–937). Perth,
Australia.
Williams, J. B. (2006). The place of the closed book, invigilated final examination in a knowledge
economy. Educational Media International,43, 2, 107–119.
Appendix 1
Survey questions
Consider each of the following statements in relation to your perceptions of a typical closed
book, invigilated exam that you have taken at some stage in your academic career. Please select
one number from 1 (strongly disagree) to 5 (strongly agree) that most closely represents
your view
1. The time period allowed (eg, 3 hours) was about right.
2. Attending a physical location at a certain time was convenient.
3. The exam was in keeping with the pedagogy (the style of content delivery, assign-
ments, etc) employed during the course.
4. The exam produced high-quality learning outcomes.
5. The exam was intellectually challenging.
6. The exam format suited my learning style.
7. The exam format was relevant to my business/professional education.
8. The content of the exam was engaging.
9. The structure of the exam meant students could cheat.
Consider each of the following statements in relation to your perceptions of a typical
open book, open web (OBOW) exam that you have taken at U21Global. Please select
one number from 1 (strongly disagree) to 5 (strongly agree) that most closely represents
your view
1. The time period allowed (ie, 24 hours) was about right.
2. Not attending a physical location at a certain time was convenient.
3. The exam was in keeping with the pedagogy (the style of content delivery, assign-
ments, etc) employed during the course.
4. The exam produced high-quality learning outcomes.
5. The exam was intellectually challenging.
6. The exam format suited my learning style.
7. The exam format was relevant to my business/professional education.
8. The content of the exam was engaging.
9. The structure of the exam meant students could cheat.
Efficacy of final examinations 235
© 2009 The Authors. Journal compilation © 2009 Becta.
Appendix 2
Sample survey question
9) The exam was in keeping with the pedagogy (the style of content delivery, assign-
ments, etc) employed during the course.
䊊1. Strongly disagree
䊊2. Disagree
䊊3. Neutral
䊊4. Agree
䊊5. Strongly agree
䊊Not applicable
10) Please add any other comment you feel is relevant:
Not applicable
236 British Journal of Educational Technology Vol 40 No 2 2009
© 2009 The Authors. Journal compilation © 2009 Becta.
