Content uploaded by Yvonne Alexandra Breyer
Author content
All content in this area was uploaded by Yvonne Alexandra Breyer on Aug 06, 2014
Content may be subject to copyright.
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
1
Finished publication available here:
http://www.igi-global.com/chapter/piloting-online-submission-and-online-
assessment-with-grademark/109267
Piloting Online Submission and Online
Assessment with
GradeMark
Trudy Ambler, Yvonne Breyer, Sherman Young
Macquarie University, Sydney, Australia
ABSTRACT
Online technologies are becoming ubiquitous in higher education and present both challenges and
opportunities for those involved in learning and teaching. This paper reports on the research-enhanced
implementation of electronic assessment management (EAM) within one faculty of a university in
Sydney, Australia. This research was conducted as a qualitative case study. Questionnaires were used to
investigate staff and student experiences of EAM, and the researcher’s reflective practice made it possible
to capture important details of the implementation process mediated through the researchers as
participants. The research found enormous potential in EAM implementation for cultural transformation
in learning and teaching. We argue that the move to EAM is now a viable option for universities. The
combination of a rapidly evolving higher education landscape, evidence from exploring both staff and
student experiences of engaging with EAM and the benefits which the transition offers for the
professional development of academics make the use of EAM essential for reasons of both pedagogy and
efficiency.
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
2
INTRODUCTION
The use of online technologies has become increasingly prevalent in the higher education sector.
Learning Management Systems (LMS) are supporting the delivery of unit content, lecture recordings and
electronic reading materials (Johnson, Adams, Cummins, 2012; Reigeluth, Watson, Watson, Dutta, Chen
& Powell, 2008), and there are more recent explorations into technology-enhanced teaching, for example
through virtual worlds (Dalgarno, Lee, Carlson, Gregory & Tynan, 2011) or networked games-based
learning (Connolly & Stansfield, 2009). Still to be fully implemented in many tertiary institutions is the
use of online technologies to streamline the submission, marking and return of traditional written
assignments –activities which are central to learning and teaching in higher education. In this paper, we
argue that the move to electronic assessment management (EAM) is now a viable option; universities
need to examine how it might enhance the student and staff experience. We will highlight the potential
this move holds beyond the transition of submission mode from hard copy to the screen, as well as
identify the challenges of such a move. A starting point is to scope the current higher education landscape
and identify particular features making the move to EAM attractive. Following is a description of a
research project that explored the introduction of EAM within the Arts Faculty at Macquarie University in
Sydney, Australia. Here we specifically highlight the implementation process, as this is still an aspect
largely missing from the EAM professional literature. Finally, we offer our reflections on the use of
EAM, noting key issues that emerged during the research that may provide helpful insights for any
practitioners preparing for the move from paper to screen.
BACKGROUND
As is common with other information industries, the university sector is facing challenges
brought about by the rapid development and diffusion of internet technologies. As content becomes
readily available for little cost, and disruptive global competitors emerge in the form of free massive,
open, online courses (MOOCs), there is increasing pressure for universities to articulate their
distinctiveness through pedagogy and to ensure the efficiency of their administrative processes. The
implementation of EAM provides an opportunity to engage with both pedagogy and administration.
As the University of the Future report (Ernst & Young, 2012) suggests, “universities have ample
scope to streamline their business and operations” (p. 17). Whilst there are many ways to effect such
streamlining, the imperatives are not simply to lower costs. Indeed, the more useful driver is that of
improving the student focus – and to ensure that traditionally cumbersome, bureaucratic, administrative
processes are displaced by the types of responsive (online) procedures more commonly experienced by
the so-called ‘digital natives’ (Prensky, 2001). This is also a group of students who “expect to be able to
work, learn, and study whenever and wherever they want to” (Johnson, Adams & Cummins, 2012, p. 4;
see also Gosper, Malfroy & McKenzie, 2013) – a cohort which requires a more flexible approach to
providing student services and support. Government legislation to broaden access to higher education to a
more diverse student body (Bradley, Noonan, Nugent & Scales, 2008) and the removal of restrictions
around enrolment numbers have transformed and, at the same time, increased the student population. One
core area of these activities is the management and assessment of written student assignments, which have
traditionally been delivered and marked on paper. Managing assignments in hard copy requires
substantial handling by administrative staff who receive assignments from students, collate them, record
the submissions and distribute assignments to markers. In the Faculty, this is handled centrally for several
thousand undergraduate students, all submitting a number of pieces of written work for each unit of study
undertaken in each semester. Requiring students to submit via hard copy means that students need to print
out and hand-deliver or post assignments. As a result, often students need to travel to campus merely to
hand in an assignment. Due to a general increase in student numbers, the task of handling assignments in
hard copy has become more labour-intensive. Whilst older models of Oxbridge-style tutorials still exist,
they are increasingly subsidised by the reality of mass tertiary education; class sizes regularly run into the
hundreds, and management of learning activities and assessment needs to respond to this shift. Local
sustainability policies have impacted on the way universities conduct their everyday business. In
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
3
particular, they are required to ensure that their core business operations are sustainable and
environmentally friendly (Sharp, 2009). At Macquarie University, all these factors converged in 2010,
and the Faculty of Arts took the opportunity to rethink its learning and teaching approach. In response to
the drivers for change described above, the Faculty of Arts at Macquarie University ran an EAM pilot in
2011.
Prior to the project commencing, submission of hardcopy assignments in the Faculty of Arts was
via the Faculty student centre office by way of large deposit boxes. There was no receipt of assignment
submission for students available. External students submitted hardcopy assignments either in person or
via mail to the University’s Centre of Open Education (COE), which did provide submission receipts.
Some convenors accepted electronic submissions on an ad hoc basis. This was handled through the
assignment drop box in the LMS, via email, through COE, or in some cases through Turnitin – which was
in common use for detecting plagiarism. Some staff used a combination of Turnitin and hardcopy
submission, in effect requiring students to submit their paper twice. There were a number of issues around
each of these processes. Hardcopies could be easily lost during the submission and distribution process
and, because a receipt is not issued, it is not possible to track the original submission. Whilst the Faculty
office kept a record of submissions – and a system of barcoded cover sheets made this process less
onerous – the sheer number of assignments processed resulted in inevitable errors of record-keeping.
Similarly, papers submitted electronically either via the LMS drop box or email were not officially
receipted. So, hard copies and electronic submissions required a complex administrative workflow
involving multiple handling and manual distribution. And, of course, travel to campus was required for
both staff and students when hard copies were submitted and subsequently collected for marking.
The marking of hardcopy student assignments involves providing feedback via hand-written
comments, using an attached rubric or marking grid or a combination of the two. Providing hand-written
comments on hard copies entails some constraints: comments, particularly for large cohorts, are often
short by necessity, and can be illegible and inconsistent. Additionally, because hardcopy assignments are
returned to students, any comments written on hard copies are lost for future reference – making dealing
with grade appeals processes less than ideal. Electronic submissions are either marked via a ‘track
changes’ functionality in a word processor or via pasting previously saved comments, attaching a rubric
or marking grid document, or a combination of both. Marking electronically in a word processor involves
downloading and storing the original assignment locally, inserting comments via track-changes or pasting
comments stored in a separate document. These processes can be time-consuming and prone to manual
errors. Administratively speaking, marking hard copies and electronic copies in these ways require the
marker to enter marks both on the paper and then again in a separate document and the process of
collating and moderating marks is onerous, particularly for large cohorts.
The Faculty policy is to require hardcopy assignments to be returned in classes. As well as
minimising the administrative load on the Arts Student Centre, this gives tutors the opportunity to provide
face-to-face feedback both individually and as part of a more general discussion. However, tutorial time is
reduced when handing out papers, and students not present on the day experience a delay in getting
valuable feedback. Equally, returning electronic assignments by uploading individual pieces of work to
the LMS or attaching them to emails is time consuming and onerous for the staff member. Public, mass
distribution of assignments is not viable – institutional concerns for privacy make such an option
problematic.
In summary, at the beginning of the project, whilst some academic staff were managing
assignments online (e.g., via email attachments or through the in-built assignment drop box in the LMS),
the majority of assignments were submitted and returned as hard copy. Convenors who used electronic
marking did not experience the benefit of a properly integrated system for submission, assessment and
return of assignments.
RESEARCH OUTCOMES
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
4
The purpose of this research project was to explore the potential of a fully integrated EAM
system, by piloting an online marking system for a small group of self-selected unit convenors. For this
research, the following outcomes were identified:
(i) Explore staff and student perspectives on EAM with a particular focus on workload reduction,
increased flexibility for staff and potential improvement of the student experience due to timely,
legible and transparent feedback;
(ii) Identify potential issues/obstacles connected to administrative as well as academic aspects of
EAM both from the staff and student perspective;
(iii) Evaluate the potential of integrated solutions technology for learning and teaching, i.e.,
assignment submission, assessment and feedback; and
(iv) Evaluate the potential of Turnitin/GradeMark as a possible solution for EAM at Macquarie
University.
TECHNOLOGY OR TECHNOLOGIES
The research team identified the following as prerequisites for the choice of technology:
• submission and automated return of electronic student assignments,
• receipting of student submissions,
• integrated plagiarism checking,
• feedback and general comments functionality,
• efficient marks entry, and
• a capacity to include rubrics.
The option to build a software solution in-house was not immediately available. This was due to a lack of
funding and the ongoing ability to maintain and further develop such a system in the future. As a result, a
third-party solution was required. The software selected for the project was the GradeMark® tool within
Turnitin; this selection was based on the best research evidence available at the commencement of the
project. Literature on EAM identified this software to be an “innovative assessment and feedback tool
[…] which benefits both academics and students pedagogically” (Chew & Price, 2010, p. 687).
Henderson (2008) supported this view and also found that “in terms of ease of use, speed and targeted
comments, GradeMark represented an impressive electronic means for marking that eliminates repetition,
organises the task and automatically enters marks” (p. 11.2). In a review of 36 online marking and
feedback systems, Shortis and Burrows (2009) concluded that on the basis of their review followed by a
user analysis test Turnitin/GradeMark were considered to be among “the most favoured systems at that
time” (p. 310). This view is echoed in an update of their review two years later (Burrows & Shortis,
2011). Additionally, for the pilot, the choice of third-party software was quite straightforward. The
plagiarism detection software Turnitin had already been employed campus-wide for some years to ensure
high standards of academic integrity. As a result, staff and students were already familiar with this
system. It was decided to purchase a trial licence for GradeMark, the fully integrated extension for online
assessment of written work in Turnitin.
Turnitin has two core functionalities: firstly, the OriginalityCheck tool compares student submissions
with large online text databases in order to check for similarities; and secondly, the GradeMark
integration features a sophisticated feedback system for online marking. OriginalityCheck has the
capacity to identify and highlight text which matches sources found online. This helps the teacher to
determine whether there is evidence of plagiarism and to initiate an educative process to ensure students
maintain high levels of academic integrity. GradeMark includes functionality for providing assessment
feedback in three ways: firstly, teachers can insert detailed written comments and save these for future
use; secondly, a general comment can be added (this can also be delivered in the form of a voice
recording); thirdly, a rubric can be attached to the assignment, and student performance can be indicated
on the rubric (qualitative, quantitative or customised rubrics can be used).
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
5
The LMS used at Macquarie is Moodle and staff access Turnitin/GradeMark via the direct
integration module provided by Turnitin. This integration supports core functionality in Moodle, such as
the creation of Turnitin assignments in the native Moodle environment, the transfer of marks to the Grade
Book and the ‘Groups’ functionality. The latter is particularly useful when working with large student
cohorts and multiple markers. The integration is essential as this makes the management and handling of
student submissions a streamlined and largely automated process controlled by the software. The teacher
sets up the Turnitin assignment as a link within the LMS and determines the due date and the post date.
The latter is the projected date on which students will receive their marked assignments; GradeMark
automatically releases them on that day. Teachers can change this date as required at any time.
Students submit via the assignment link in Moodle by uploading their document (multiple common
file types are accepted). The system generates a unique paper identification number (ID) and
automatically sends a receipt with that ID with a time stamp of submission to the student. The teacher can
view the assignment, including student name, paper title, time stamp, similarity index, and grade in the
submission inbox. Once the teacher has finalised the marking process, the assignments can either be
released directly by adjusting the post date or they will be automatically released on the pre-determined
release date. Students can then view their marked assignments by following the initial assignment link.
The assignment can be viewed either in the interactive view online or students can download a Portable
Document Format (PDF) file of the marked assignment for future reference. This PDF file includes a
grade report in the form of endnotes, the general comments and the final mark on the paper. In the staff
interface, a visual indicator confirms to the marker whether or not a student has retrieved the marked
assignment – and the feedback. The integration of Turnitin in the LMS communicates the mark entered
on the original paper automatically to the gradebook of the LMS. This means that marks entry need only
occur once – on the assignment itself.
Staff can view assignments in Turnitin via the Turnitin Document Viewer in the web browser
which operates across all operating systems and is largely platform-independent. No downloading or
uploading of assignment files is required; however, access to assignments can only be made with an
internet connection. Importantly, it is possible to view the original submission file at all times as the
marking process does not overwrite it. All teachers also receive an account to access Turnitin via the
official Turnitin website. More functionality, for example grade statistics, is available via this avenue.
Currently, Turnitin integrates with more than 50 LMSs.
CASE STUDY
This research project was funded by the Macquarie University Emerging Technology Grant
scheme, and the purpose of the case study was to explore EAM of written student assignments.
Assessment can take many forms but in this research, we are referring to written assignments that have
traditionally been submitted for marking via hard copy. For this type of assignment (usually annotated
bibliographies, essays and the like) there are three main components to the EAM process: submission,
marking and return. As stated previously prior to the trial of GradeMark, practices in the Faculty around
assessment were managed either via hard copy, electronically or a combination of the two. There was no
campus-wide online assessment management in place, and access to a streamlined EAM solution was not
available.
The research reported here took place in the Faculty of Arts (Macquarie University, Sydney)
during 2011 and 2012 and ethics approval to conduct this study was sought and granted by the Human
Research Ethics Committee at Macquarie University. The study was structured into a mixed methods,
two-phased pilot. This type of approach was selected as it provided the opportunity to collect both
qualitative and quantitative information about staff and student experiences of EAM. Integral to the
qualitative aspect of the data collection, staff reflective practice was also used. It is argued by Adler
(1993, p. 160) that any conception of research should include reflective inquiry; she comments:
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
6
Research as reflection on experience can enable us to become more aware, to see again that
which we take for granted, to find the significant in the insignificant (van Manen, 1990).
Reflection on experience is, therefore, a valid basis for practical action, a way of informing our
practice.
The literature and our own experience as researchers and teachers reinforced our view that reflection is a
necessary part of teaching, learning and research (Adler, 1993; Boud, Cohen, & Walker, 1993; Darling-
Hammond, 1997; Dewey, 1916; Kolb, 1984; McAlpine, Weston, Berthiaume, Fairbank-Roch, & Owen,
2004; Schon, 1987, 1991; Smyth, 1992). In this research project, reflective practice was used as a method
to interrogate our experiences (as participants integral to the process) in a scholarly way informed by both
our teaching experience and “educational research or theory” (Kreber, 2005, p. 326). Research
discussions with participants about the data proved to be a dynamic medium that we used to develop our
understandings about EAM. Through a process of recollecting and describing past experiences of
particular activities and events in the research, we enhanced our own learning through a more sustained
and careful consideration of those experiences.
The first phase of the research took place in 2011 and involved a total of eight teaching staff in
eight units of study. These staff had responded to a request for volunteers. Student cohorts of the
individual units ranged from 10 to approximately 500 students. Staff participants in the first phase of the
pilot received individual training on the use of GradeMark from the research team in one-hour sessions.
Student support was provided by staff responsible for the units involved. At the end of 2011, staff and
students were invited to participate in an online questionnaire. The questions focussed on staff and student
experiences around EAM. Some of the questions were guided by a pre-determined set of questions which
needed to be included in any evaluation of projects funded through the Emerging Technologies internal
grant scheme. The questionnaires for staff and students differed somewhat in the questions (e.g., staff
questions focussed on the setup of assignments) but for both groups participation was optional and
anonymous and both were made up of closed, semi-open and open-ended questions. The online service
SurveyMonkey was used to distribute the questionnaires online, and it also supported the data analysis
through its graphing functions.
STAFF EXPERIENCES OF USING TURNITIN/GRADEMARK
On completion of using the software for assignment submission, marking, grading and return of
work, all convenors who took part in the pilot in 2011 were invited to participate in the online survey that
explored their experiences of using the software. All participants (8 staff) responded and all indicated a
preference for online submission over paper submission. The capacity of the software to enhance
efficiency and effectiveness (Allan & Bentley, 2012; Heinrich, Milne, Ramsay & Morrison, 2009) of
assignment management (i.e., receiving, viewing, managing, marking, and returning assignments) was
rated either highly satisfactory or satisfactory by the participants. The positive environmental impact was
a feature that respondents considered to be significantly improved as it was determined to be “more
sustainable, faster, more reliable, more efficient to grade and provide feedback.” Staff also identified that
existing familiarity with Turnitin helped them in the transition to online assessment. One participant
summarised their experience with the software as follows: “It is efficiently a ‘one stop shop’ and the
quality of information/feedback offered to students is timely and targeted.”
Respondents rated their satisfaction with the software in terms of its reliability in learning and
teaching practice, layout on the computer screen and general ease of use as highly satisfactory and
satisfactory. The main issue identified by participants was the speed and connectivity to the internet when
using the software. It was noted during the pilot that the speed of their internet connection had an impact
on the functionality of the software, and one staff member in their comments stated that, “moving back
and forth between assignments and inserting comments” was problematic when the connection was slow.
Academics strongly agreed or agreed that administrative processes for the submission and return of
student work were more effective and timely within the online environment. A comment was made that,
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
7
“[u]sing GradeMark is faster as most of us can type faster than we can write. Further, my tutors and I
agreed that GradeMark readily lends itself to standardisation of marking…” Participants identified that
the software offered a very systematic process for identifying plagiarism, and this was useful as it created
opportunities for educating students (Cohen, 2010) about this essential aspect of academia. Online
marking also allowed staff to provide more detailed and targeted feedback (Henderson, 2008). This
resulted from the multi-functional feedback options in the software that enables rubrics to be used,
general comments to be scripted and in-text comments, deletions and additions to be inserted into the text.
One staff member commented, “… I like to employ ongoing assessment in my unit. This [GradeMark]
provides a faster mechanism for collecting, grading and returning assignments with feedback.”
As part of the environmental impact analysis, staff respondents were asked whether there were any
issues that arose in relation to quality assurance and compliance with regulatory frameworks. This
included issues related to accessibility needs for students, information management, confidentiality,
intellectual property, copyright, and quality assurance. From a quality assurance perspective, the software
was rated very highly. However, one staff member pointed out that it is hard to see what students see, as a
student view is not available. Staff respondents pointed out a few technical improvements that could be
made to the software such as the development of an iPad application but when asked if they would
‘recommend the use of GradeMark to colleagues?’ all replied yes. Based on the success of the pilot, at the
start of 2012 the Faculty of Arts made a decision to offer GradeMark to any convenor that wished to use
it in their teaching. In Session 1, 2012, 45 units used GradeMark for online assessment in Arts. That
number nearly doubled for Session 2, and some Departments indicated their intent to move all their
assessment online for the following year. The success of the pilot in the Faculty of Arts lead to an upgrade
of the GradeMark license to an institutional licence in 2012. Campus-wide, the number of student papers
marked with GradeMark increased from 1,876 in 2011 to 21,505 in 2012. This is strong evidence of the
widespread adoption of this tool across the institution.
It is also evidence of the inherent scalability of the system. Whilst there has been a need to assess
support issues and the requirement for staff professional development, the shift from the pilot to ‘common
use’ was relatively straightforward. It is beyond the scope of this study to discuss that transition in detail,
but scaling to a larger number of users has not been problematic. Arguably, the choice of technology
(using a third party provider with service level requirements, rather than developing or hosting a solution
in house) has helped in this regard. Whilst there have been technical issues with Grademark, this has been
largely due to problems that have arisen as a consequence of specific software integration or local
network problems, rather than as a result of the number of users. Of course, higher levels of usage means
that there is a greater risk to the institution in the event of problems – something which needs to be
managed.
STUDENT EXPERIENCES OF USING TURNITIN/GRADEMARK
Students were provided with information about submitting their assignments but received no
further introduction to using Turnitin/GradeMark. Having taken part in the pilot, they were invited to
respond to the online survey. A complicating factor in reaching students was that the ethical framework
for this study required that the invitation to undertake the survey could only be sent out after the
completion of the unit and release of final grades. Participation in the survey was voluntary and
anonymous, and 51 students from six units of study elected to respond.
With regard to accessing technology, the majority of student respondents indicated that they had
access to a computer (95.7%) and to broadband internet (93.5%) as well as a printer (78.3%). One student
indicated that they “… would prefer not to use [my] the printer at all and conserve paper and ink.”
Students used the software largely successfully with only a small minority reporting technical issues
when submitting their assignments. Nearly two-thirds of the students (64.4%) preferred online submission
while 11.1% had no preference. Almost all student respondents strongly agreed or agreed that online
submission is preferable because it saves time and cost, reduces travel to campus and printing; a view
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
8
captured in the comments made by a student who stated, “I live on the Central Coast so obviously it was
great that I could submit assignments online, and not waste 4 hours traveling to and from university only
to hand in a hard copy.” Retrieving assignments online meant that students had instant access to their
marks, it was an environmentally-friendly (no need for paper and printers) and also cost-effective
alternative for them: “I live over an hour from campus, so it is such a great alternative...especially seeing
petrol is costing so much!” Students believed it was a convenient way to store past assignments and to
view their mark and assessment feedback.
In relation to the student experience, there is strong evidence that feedback in higher education is
seen to be inadequate and this “… continues to be an issue” (James, Krause & Jennings, 2010, p. 57). It is
noticeable that “[o]nly one-third of students believe that staff usually give them helpful feedback on their
progress” (ibid.). It is therefore an encouraging result that 75% of student respondents in this research
reported that they had viewed the in-text and general comments made on their assignments. Furthermore,
82.5% found that “electronic feedback is always legible” and 20% reported that “electronic feedback
appeared to be more detailed and comments were longer” (as found in Carless, Salter, Yang & Lam,
2010). As one student commented: “Often the manual feedback on papers is difficult to read and
scrawled on occasions. I found the Turnitin results excellent.” Over half of the students (53.7%) preferred
electronic over hand-written feedback while 17.1% had no preference. Most students (90%) indicated that
they did not encounter any difficulties in relation to receiving their assignments back after marking, with
a small percentage suggesting that clearer instructions should be provided to explain the functionality of
the software when accessing marked work. A small improvement can be found in 26% of the students
reporting to have spent more time than usual on reading the electronic feedback provided on their
assignments. In total 75% of all student respondents agreed they would like to see online submission and
online marking of student assignments more widely used at Macquarie University.
REFLECTIONS
The reflections in this section arise from the authors’ critical engagement with the literature, the
results of the research and their own experience of participating in the study. It is clear from this project
that there are some key points to consider for those planning to introduce technological innovation in
higher education, specifically focused around assessment. There are issues related to change management,
planning and designing opportunities for professional development, performance of related technologies;
and future research opportunities.
Issues of change management
Tertiary educational institutions are faced with managing large cohorts of students, staff from
diverse backgrounds with differing professional capacities and a pervasive regulatory policy environment
to which they must respond. Additionally, change within any work environment is often met with
suspicion and resistance. However, there has rarely been an agent for change as powerful and developing
as rapidly as the widespread introduction of online technologies. As a result of this, the project team is
aware that ‘change is the new default’ and that, rather than assume that change will be followed by long
periods of stability, any twenty-first century workforce needs to engage with the reality of constant
change. In that context, successfully managing the ongoing challenges presented by technological
innovation is critical for universities.
There are three aspects relevant to change management that we want to highlight. Firstly, the
move to EAM was precipitated by a range of external forces such as a reduction in administrative
resources, increasing student numbers and a general move by the institution to provide learning and
teaching independent of location and time. When faced with this situation, the response from the Faculty
was to initiate a pilot driven by top-down leadership (e.g., directive from Faculty Board) and a bottom-up
approach of early adopters (e.g., academics who volunteered to participate in the pilot and who
subsequently promoted the use of EAM). This situation prompted the authors to question how well
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
9
universities are equipped to become proactive adopters and participants prepared to integrate new
technologies into learning and teaching as opposed to passive recipients who are forced into change by
external forces. What role does leadership have to play in this? Executive leaders need to have a vision
and direct faculty with initiatives to drive change. Based on our experience, leaders need to ensure that
they are properly informed about the affordances of new technologies or, alternatively, employ staff with
the expertise to research and advise them in their decision-making process.
Secondly, assessment as an aspect of learning and teaching is high-risk and high-impact both for
staff and for students. Any disruption (perceived or real) to the process of receiving, managing, storing,
marking and returning assignments needs be minimised at all cost. Throughout this research, it became
clear that any technology related to assessment has that critical risk factor. Thus, leaders need to
implement and maintain risk management strategies. To avoid disruptions, it is vital that the technology is
reliable, readily accessible, and user-friendly and meets all required specifications related to the EAM
process. Professional development for staff is imperative in order to ensure correct use of the technology.
Additionally, students need to be provided with adequate guidance in order to become familiar with the
technology. This includes raising awareness on all aspects related to EAM (e.g., archiving copies of past
assignments for future reference and creating backups, etc.).
In the event of a critical incident, it is important to have measures in place in order to respond
swiftly and effectively. One of the conditions for this is buy-in from all relevant stakeholders in the
institution, specifically key leaders from both the academic and professional groups. In the case of this
research, the University’s Learning & Teaching Centre (LTC) which supports all learning and teaching
technologies across the university agreed to support the use of Turnitin/GradeMark after the successful
conclusion of the pilot. This was pivotal in the decision by the Faculty to expand the trial and ultimately
led to the adoption of this technology across all faculties at the university. A special interest group
(Turnitin User Group) was also established. This group is made up of representatives from faculties as
well as members from the LTC. Its role is to address issues of risk management, to collect feedback from
end-users in order to inform training sessions and to disseminate research developments occurring in the
broader Turnitin/GradeMark community.
The present case study was framed as a research project and this decision had some tangible
benefits. Managing the project as a research study enabled the authors to draw on existing literature in
order to inform any decisions related to the pilot which included the choice, feasibility and functionality
of the software. It also offered evidence for those in leadership positions to substantiate their decisions
regarding EAM when communicating with the broader academic group in the Faculty (such as the
Faculty Learning and Teaching Committee and the Faculty Board). All research projects involving
humans are required to gain approval from the Human Research Ethics Committee. Ethics approval was
granted early in the project, which permitted the collection of data from staff and students during the
implementation phase. This data was subsequently available for use in publications in order to
disseminate the findings to the wider educational community.
Technology as a catalyst for professional learning
With the ever-increasing pace of technological change, it is crucial for university staff to adapt
fast and develop skills and strategies to harness new technologies as they emerge. Throughout this
research it became apparent that preparing academics to engage with EAM requires a multifaceted
approach due to their level of confidence in technology as well as their motivation and their general
availability to participate. Academic staff have varying Information Technology (IT) skill sets which are
acquired through a mix of professional training and personal experience. In our experience, the level of IT
proficiency demonstrated by staff is frequently linked to their willingness to engage with new
technologies. Diverse work patterns that involve full-time ongoing, part-time, sessional staff and some
staff who are remote from the university and do not attend campus (such as those teaching in the Open
University programs) also meant that staff had limited capacity to engage with training sessions. In
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
10
response to these drivers, we made the decision to create a program of professional development that was
flexible and offered in various modes. In collaboration with the LTC, online resources known as ‘Quick
Guides’ were created which staff could access through the university website and use as independent
learners from any location. The Faculty also invested in a bank of 30 iPads so that anyone could access
the online training resources flexibly. Face-to-face workshops for staff within individual departments
were organised, and these were offered either by request or as scheduled events throughout the semester.
Additionally, staff could arrange bespoke one-on-one sessions with the researchers if they wished to focus
on something specific to their unit of study. Group sessions were offered to unit convenors with large
groups of tutors. This provided the opportunity to train tutors and convenors in the new system and at the
same time discuss the marking process and rubric within that group.
We are reminded by Guskey (2002, p. 386) in his work looking at professional development and
teacher change that:
[l]ike practitioners in many other fields, teachers are reluctant to adopt new practices or procedures
unless they feel sure they can make them work (Lortie, 1975). To change or to try something new
means to risk failure. […] Even when presented with evidence from the most carefully designed
experimental studies, teachers do not easily alter or discard the practices they have developed and
refined in the demanding environment of their own classrooms (Bolster, 1983).
The outcomes of this project have demonstrated that promoting teacher learning can be encouraged on a
number of levels. Our multi-faceted approach to professional development provided the opportunity to
build capacity within departments and the Faculty because it fostered incidental learning through the
sharing of practices and experiences with colleagues around the use of EAM. Those staff members who
participated in the pilot all reported in the questionnaire that they would recommend the use of
Turnitin/GradeMark to colleagues. These findings were promoted in the Faculty when the call to expand
EAM was sent to the rest of the academic staff. The work undertaken by these early adopters then became
a model for practice used to engage other staff into taking action to introduce EAM into their work.
Without this broad spectrum of opportunities for staff development increasing the capacity of academics
to engage with the benefits presented by this technology would have been limited.
A powerful aspect of the professional development in this study underpinning the bespoke, group
and incidental learning was the quality of professional conversations. In these academic-to-academic
interactions, conversations were initiated that moved beyond mere technological implementation and
instead broadened to educative discussions where it was possible to interrogate teaching practice.
Questions were generated that made it possible to unpack both pedagogical and administrative work
practices used by academics and this enabled the introduction of changes to those practices. It also
encouraged conceptual growth in understandings and the development of strategies for continued
engagement with technological innovations in learning and teaching. The learning and teaching expertise
of the research group also needs to be acknowledged in the role of staff development. The colloquial
expression ‘it is always difficult to know what you don’t know’ is very appropriate here as it was the
knowledge of skilled educators that was instrumental in leading staff to reconfigure and construct
cognitive frameworks that will help to prepare them for ongoing participation in the online learning and
teaching environment.
Technological provision
The computer hardware, software and supporting network facilities provided and owned by
universities are resources that are integral to a successful online learning and teaching experience,
specifically the move to EAM. Desktop and mobile computers such as laptops used for EAM have to be
up-to-date and fit for purpose. This involves ensuring the regular renewal of equipment so that academics
have computer screens that can cope with the display requirements needed for assessing work online.
Software must also be carefully managed. Maintaining productive relationships with software providers is
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
11
helpful when gaining and developing product knowledge and it is essential that providers have
mechanisms for responding to enquiries and considering product development ideas. The reliability and
robust quality of the network is also essential. Processing large volumes of student work and marking
online is only possible if the infrastructure is reliable and connectivity is readily available.
Affordances of technology beyond change of mode
There are affordances resulting from the use of EAM technology beyond the move from hard
copy to online assessment. One benefit of moving paper-based marking online is the occurrence of
learning data. In the case of Turnitin/GradeMark teachers have access to an interesting and valuable range
of statistical data. The analysis of such learning data has been highlighted as one of the main areas of
interest over the next few years by the Horizon report (Johnson et al., 2012). Student assignments stored
and managed electronically produce a wealth of information which the research community is only just
starting to tap into. Examples of this information are performance statistics from student cohorts or
individual students. Teachers are able to review data on how, for examples, specific students or an entire
cohort performed against specific rubric criteria or even what type of errors that were made most
frequently. Evaluations of such data can inform future teaching processes and provide opportunities to
target problematic areas. Increasing numbers in student cohorts provide challenges for teachers in terms
of keeping track of an individual’s performance; however, the availability of electronic data can be a
valuable source of information to feed forward.
CONCLUSION
The successful pilot of Turnitin/GradeMark precipitated a broader rollout of that particular EAM
technology across the Faculty. As a consequence, the practice of online submission and assessment is fast
becoming accepted practice for many individual teachers as well as whole departments within the Faculty
of Arts. Staff members have repeatedly lauded the efficiencies that the system provides, and students are
demonstrably happy with the provided feedback, both in terms of quality and responsiveness. The
administrative advantages have been clearly demonstrated through a reduced workload in the office
concerned. Additionally, the rollout of EAM has provided important opportunities for professional
development around assessment and feedback. Indeed, the project demonstrated that a technological
implementation can be an important trigger for professional learning. Conversations that commenced
around technological implementation developed into pedagogical discussions related to standards-based
assessment and the use of rubrics. The implementation of EAM was an important catalyst for these
conversations. .
It may be that the staff who volunteered for this pilot study could be considered early adopters with a
propensity for positive evaluation of new technologies. However, the pilot’s conclusions that EAM can
make a significant contribution to managing assessments have been further demonstrated by the
subsequent successful and widespread uptake of EAM in the Faculty. A recent Faculty report
demonstrated that on average 63.5% of all students in enrolled in Arts units received electronic feedback
via Grademark in the first Session of 2013.
Given that assessment is such a high stakes area, this research concludes that, whilst the move to
EAM is imperative, it is a move that needs to be well managed and planned. Further research is required
on the development of EAM systems that are responsive to the varying modes of assessment now being
completed within higher education courses; this includes outputs such as films and artifacts. More work is
also required to develop tools that make sense of the learning analytics data generated by EAM processes,
inclusive of investigating what students make of this ‘improved’ and data-rich feedback in their learning.
The importance of professional development for academics throughout technological innovation was
paramount, and more research needs to be initiated to determine if staff can transfer newly developed
conceptual understandings into different contexts where online technologies are introduced.
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
12
REFERENCES
Adler, S. (1993). Teacher Education: Research as Reflective Practice. Teacher and Teacher Education,
9(2), 159-167. Retrieved from http://dx.doi.org/10.1016/0742-051X(93)90051-H
Allan, R., & Bentley, S. (2012). Feedback mechanisms: Efficient and effective use of technology or a
waste of time and effort? In STEM Annual Conference, 2012, 12-13 April 2012, Imperial College,
London. Retrieved from http://eprints.hud.ac.uk/13442
Boud, D., Cohen, R., & Walker, D. (1993). Using experience for learning. Buckingham: SRHE Open
University Press.
Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of Australian Higher Education.
Retrieved from http://www.innovation.gov.au/HigherEducation/Documents/Review/PDF/
Higher%20Education%20Review_one%20document_02.pdf
Burrows, S., & Shortis, M. (2011). An Evaluation of Semi-automated, Collaborative Marking and
Feedback Systems: Academic Staff Perspectives. Australasian Journal of Educational
Technology, 27(7), 1135-1154. Retrieved from
http://www.ascilite.org.au/ajet/ajet27/burrows.html.
Carless, D., Salter, D., Yang, M., & Lam, J. (2010). Developing Sustainable Feedback Practices. Studies
in Higher Education, 36(4), 395-407.
Chew, E., & Price, T. (2010). Online originality checking and online assessment – an extension of
academics or disruption for academics. Paper presented at the Proceedings of the 18th
International Conference on Computers in Education, Putrajaya, Malaysia, (pp. 683-687).
Retrieved from http://www.icce2010.upm.edu.my/papers/c6/short%20paper/C6SP43.pdf
Cohen, J. (2010). Using Turnitin as a Formative Writing Tool. Journal of Learning
Development in Higher Education, Issue 2, February. Retrieved from
http://www.aldinhe.ac.uk/ojs/index.php?journal=jldhe&page=article&op=view&path%5B%5D=
22
Connolly, T., & Stansfield, M. (2009). Games-based E-learning: Implications and Challenges for Higher
Education and Training. In I. Lee (Ed.), Electronic Business: Concepts, Methodologies, Tools,
and Applications (pp. 910-924). Hershey, PA: Information Science Reference.
Dalgarno, B., Lee, M. J. W., Carlson, L., Gregory, S., & Tynan, B. (2011). An Australian and New
Zealand Scoping Study on the Use of 3D Immersive Virtual Worlds in Higher Education.
Australasian Journal of Educational Technology, 27(1), 1-15. Retrieved from
http://ascilite.org.au/ajet/ajet27/dalgarno.html.
Darling Hammond, L. (1997). The right to learn: A blueprint for creating schools that work. San
Fransisco: Jossey-Bass.
Dewey, J. (1916). Democracy and Education: An introduction to the philosophy of education. New York:
MacMillan.
Ernst & Young (2012). University of the Future. A Thousand Year Old Industry on the Cusp of Profound
Change. Retrieved from http://www.ey.com/AU/en/Industries/Government---Public-
Sector/UOF_University-of-the-future
Gosper, M., Malfroy, J, & McKenzie, J. (2013). Students' Experiences and Expectations of Technologies:
An Australian Study Designed to Inform Planning and Development Decisions. Australian
Journal of Educational Technology, 29(2), 268-282. Retrieved from
http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/127.
Guskey, T. R. (2002). Professional Development and Teacher Change. Teachers and Teaching: Theory
and Practice, 8(3/4), 381-391.
Heinrich, E., Milne, J., Ramsey, A. & Morrison, D. (2009). Recommendations for the Use of E-tools for
Improvements Around Assignment Marking Quality. Assessment and Evaluation in Higher
Education, 34(4), 469-479. doi: 10.1080/02602930802071122
Cases on Assessment in Scenario and Game-based Virtual Worlds in Higher Education
13
Henderson, P. (2008). Electronic Grading and Marking: A Note on Turnitin’s Grademark Function.
History Australia, 5(1), 11.1-11.2. Retrieved from
http://journals.publishing.monash.edu/ojs/index.php/ha/article/view/275/288
James, R., Krause, K., & Jennings, C. (2010). The First Year Experience in Australian Universities:
Findings from 1994 to 2009. Centre for the Study of Higher Education, University of Melbourne.
Retrieved from
http://www.cshe.unimelb.edu.au/research/experience/docs/FYE_Report_1994_to_2009.pdf
Johnson, L., Adams, S., & Cummins, M. (2012). The NMC Horizon Report: 2012 Higher Education
Edition. Retrieved from http://www.nmc.org/horizon-project/horizon-reports/horizon-report-
higher-ed-edition
Kreber, C. 2005: Charting a Critical Course on the Scholarship of University Teaching Movement,
Studies in Higher Education, 30(4), 389-405.
Kolb, D. A. (1984). Experiential learning: Experience as a source of learning development. New Jersey:
Prentice Hall.
McAlpine, L., Weston, C., Berthiaume, D., Fairbank-Roch, G., & Owen, M. (2004). Reflection on
Teaching: Types and Goals of Reflection. Educational Research and Evaluation, 10(4-6), 337-
363.
Palmer, S. (2005). An Evaluation of On-line Assignment Submission, Marking and Return. Journal of
Educational Technology Systems, 34(1), 57-67. Retrieved from
http://dro.deakin.edu.au/eserv/DU:30004043/palmer-evaluationofonline-pub-2006.pdf
Prensky, M. (2001). Digital Natives, Digital Immigrants, On the Horizon, 9(5), 1-6. Retrieved from
http://www.marcprensky.com/writing/Prensky%20-
%20Digital%20Natives,%20Digital%20Immigrants%20-%20Part1.pdf
Reigeluth, C., Watson, W., Watson, S., Dutta, P., Chen, C., & Powell, N. (2008, November-December).
Roles for Technology in the Information-age Paradigm of Education: Learning Management
Systems. Educational Technology, 48(6), 32-39. Retrieved from
http://cardinalscholar.bsu.edu/bitstream/123456789/194511/1/LMS.pdf
Schon, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.
Schon, D. (1991). The reflective turn: Case studies in and on educational practice. New York: Teachers
Press Columbia University.
Sharp, L. (2009). Higher Education: The Quest for the Sustainable Campus (editorial). Sustainability:
Science, Practice & Policy, 5(1), 1-8. Retrieved from
http://sspp.proquest.com/archives/vol5iss1/editorial.sharp.html
Shortis, M., & Burrows, S., 2009. A review of the status of online, semi-automated marking and feedback
systems. In Milton, J., Hall, C, Lang, J., Allan, G. and Nomikoudis, N. (Eds), Proceedings, ATN
Assessment Conference 2009 (pp. 302-312). RMIT University, Melbourne, Australia. Retrieved
from
http://emedia.rmit.edu.au/conferences/index.php/ATNAC/ATNAC09/paper/viewFile/205/46
Smyth, J. (1992). Teachers’ Work and the Politics of Reflection. American Educational Research
Journal, 29(2), 267-300. Retrieved from http://www.jstor.org/stable/1163369
