Peer assessment of assignments-the USQ experience

Abstract

High quality and timely feedback on assignments is considered imperative in the learning process. Providing feedback on an assignment requires considerable time and effort on the academic's part. Yet, the effectiveness of feedback depends largely on the student's decision to both absorb and apply it. In fact, anecdotal evidence suggests that most students give insufficient attention to reviewing assessment feedback irrespective of its quality, quantity, and/or timeliness. This suggests the feedback mechanism of assessment items has questionable effectiveness in student learning. Peer assessment of assignments is one system that could potentially improve the learning effectiveness for students. In this system, each student is required, as part of assessment, to review and comment on peer's assignments. This system has been trialled at USQ in a 2009 course with a large distance student cohort. Here the authors describe the steps involved in peer assessment, summarise the work in progress, and discuss preliminary results. The authors also highlight the importance of the peer assessment system in improving students' learning at an institution with a large distance student cohort. This paper reports only the initial part of the study since final results were not to hand at the time of the preparation of this paper.

Full text

Page 1

Basnet et al., Peer assessment of assignments – USQ experience

Proceedings of the 2009 AaeE Conference, Adelaide, Copyright © B. Basnet, L. Brodie, and J. Worden 2009
Peer assessment of assignments – the USQ experience

Badri Basnet
University of Southern Queensland, Toowoomba, QLD. Australia
Badri.Basnet@usq.edu.au
Lyn Brodie
University of Southern Queensland, Toowoomba, QLD. Australia
Lyn.Brodie@usq.edu.au
John Worden
University of Southern Queensland, Toowoomba, QLD. Australia
John.Worden@usq.edu.au

Abstract: High quality and timely feedback on assignments is considered imperative in
the learning process. Providing feedback on an assignment requires considerable time
and effort on the academic’s part. Yet, the effectiveness of feedback depends largely on
the student’s decision to both absorb and apply it. In fact, anecdotal evidence suggests
that most students give insufficient attention to reviewing assessment feedback
irrespective of its quality, quantity, and/or timeliness. This suggests the feedback
mechanism of assessment items has questionable effectiveness in student learning. Peer
assessment of assignments is one system that could potentially improve the learning
effectiveness for students. In this system, each student is required, as part of assessment,
to review and comment on peer’s assignments. This system has been trialled at USQ in a
2009 course with a large distance student cohort. Here the authors describe the steps
involved in peer assessment, summarise the work in progress, and discuss preliminary
results. The authors also highlight the importance of the peer assessment system in
improving students’ learning at an institution with a large distance student cohort. This
paper reports only the initial part of the study since final results were not to hand at the
time of the preparation of this paper.

Introduction
Assessment refers to tasks assigned to students by their course instructor that are to be completed
outside of the classroom setting within a stipulated time. Written assignments accompanied by a grade
are often used as assessment in higher education institutions. Assignments are designed to reinforce
and test understanding of the theoretical concepts covered in the course while linking those concepts
with their practical applications.
Conventionally, assignments submitted by students are marked by an assessor within a given time
period and returned to the student with feedback. The purpose of the feedback is to help students to
improve their future work while passing judgement on the quality of their current work. Therefore,
timely and well-focused assignment feedback is meant to greatly enhance student-learning experiences
(Brown, Race & Rust 1995). There are different ways of providing assignment feedback. The common
method used in most higher education institutions includes hand written comments jotted on the
relevant assignment pages of a student’s individual work. A marking rubric that indicates the level of
achievement against predetermined criteria could also be used in addition to, or instead of, the hand
written annotations on student submissions.
So, assignment feedback is there to help students with their learning. However, anecdotal evidence
from teaching staff reveals that assignment feedback is poorly used by the majority of students. There
is also evidence that many students do not collect and/or review their marked assignment. Thus,
devising an effective method of communicating the required range of assignment answers through
better feedback mechanisms is essential to make the mastery of a knowledge area more effective.

Page 2

Basnet et al., Peer assessment of assignments – USQ experience

Proceedings of the 2009 AaeE Conference, Adelaide, Copyright © B. Basnet, L. Brodie, and J. Worden 2009
Attempts are being made at various educational institutions to overcome these lapses through
implementation of alternative marking practices such as the use of peer assessment system (e.g.
Bloxham & West, 2004; Prins et al, 2005; Draaijer & van Boxel, 2006; Loddington et al 2009; Willey
& Gardner 2009). In educational settings, it is generally agreed that receiving feedback about their
work and giving feedback about a peer’s work would provide students with an insight into their own
efforts while instigating the feeling of belonging to a learning community. Therefore, a peer
assessment system is often used to provide students with the opportunity to read, carefully consider,
and comment on the work of their peers, while comparing with their own work. It is frequently argued
that peer assessment is a system that provides increased understanding of the learning content, helps
develop assessment and constructive criticism skills, promotes critical thinking, and allows reflection
on one’s own performance (Draaijer & van Boxel, 2006; Prins et al. 2005; Bloxham & West 2004;
Magdeline et al. 2007; Chapman, 2009). Therefore, the application of a peer assessment system is
being trialled in one of the current courses offered in both on-campus and distance modes at the
University of Southern Queensland (USQ).
Objective & Research questions
In view of the potential benefits of peer assessment, the implementation of this system is expected to
provide better learning experiences to students with improved learning outcomes. However, this
hypothesis needs to be validated for a given learning environment. Therefore, the objective of this
study is to investigate the learning effectiveness of a peer assessment system in both an on-campus and
distance learning environment to answer the following key research questions (RQ).
1. Student reaction: How would students react to the peer assessment system? Would they find this
system useful?
2. Student performance: How would students perform with the help of the peer assessment system?
Would it contribute towards improvement of their understanding of the course material?
3. Student experience: How useful would be the peer assessment system as a learning tool? Would
they find it helpful in their learning journey?
4. Student interaction: What difference would the peer assessment system make in student
interaction? Would it instigate them to interact with fellow students via a (electronic) discussion
forum?
5. Peer feedback: How useful would students perceive feedback from their peers?
6. Student suggestion: Would students recommend future use of the peer assessment system with
some modification? What changes would students suggest if any?
Research methodology
Peer assessment is a sequential process requiring completion of several major and minor events in a
logical sequence. Therefore, the research methodology covering a number of different steps has been
subdivided into three main stages – preparation, data acquisition, and data analysis stages and largely
employs an action research methodology.
Preparation stage
Course and assessment item
Geographic Information Systems (GIS1402) is a course offered at USQ. Most students enrolled in this
course are in a distance mode. In semester 2, 2009, the ratio of on-campus to distance students has
been about 1:7. Three assessment items are used in this course including a written assignment, an
online quiz, and a closed book examination. The written assignment was selected for a trial of the peer
assessment system involving voluntary participating students. Seven questions covering about one-
half of the course content form the basis of this assignment. Each question is composed of sub-
questions requiring separate short answers as guided by the marking rubrics provided. The assignment
is worth 30% of the total marks for the course with 17.5% allocated to answering assignment
questions, and 12.5% allocated to completing the peer assessment element. Student who chose not to
participate in peer assessment were offered and equivalent alternate assessment (Figure 1).

Page 3

Basnet et al., Peer assessment of assignments – USQ experience

Proceedings of the 2009 AaeE Conference, Adelaide, Copyright © B. Basnet, L. Brodie, and J. Worden 2009
Figure 1 Assessment and research process
Learning Management System (LMS)
Moodle is the LMS used to host courses on the USQStudyDesk. All USQ courses, regardless of their
delivery mode have a 'presence' in the USQStudyDesk. The USQStudyDesk did not offer the required
‘peer assessment’ functionality. Therefore, other LMS such as Electronic Peer Review by De Raadt et
al. (2005), Moodle Workshop, and Turnitin (peer review) systems were examined. Since, the Turnitin
(peer review) is specifically designed for the purpose of peer assessment it was adopted for peer
assessment in this course. Thus, both LMS (i.e. StudyDesk and Turnitin) were utilised simultaneously
(Figure 1).
All necessary resources for students to initiate the assignment such as: assignment questions and
instructions, assignment template, and marking rubrics were made available via StudyDesk for
downloading. The Turnitin system, on the other hand, was setup to accept uploaded assignments,
automatically allocate two assignments to each student for peer review, and perform peer review of
assignments using the marking rubric provided.
Assignment submission
An assignment template, allowing only the USQ student numbers to be entered, was used by students
to submit their written assignment. This was necessary to ensure anonymous peer assessment.
Students submit assignments in duplicate in electronic format with one copy via the USQ StudyDesk
for the instructor assessment and a second copy on to Turnitin for peer assessment (Figure 1).
Instructor assessment
Student answers to the assignment questions were marked by the instructor with the aid of a marking
rubric that captured the marking criteria set in the marking rubric (refer Appendix A). The marking
rubric utilised five different quality levels starting from very poor (0-30%), limited (30- 50%),
satisfactory (50-60%), good (60-80%) to excellent (80-100%). The instructor assessed the assignment
Assignment
(30% marks)
Using topic &
metric questions
based MR
Peer
assessment
Assignment
final mark
out of 30%
Using marking
rubric (MR) as
marking guide
Instructor
marking
Alternate
assessment
(12.5%)
Instructor
marking
Marking
guidelines
Research
questions
Online
quizzes
Answers to
Research
Questions
Survey
questions
Students'
responses
Students'
performances
Discussions
on peer
assessment
Turnitin
assignment
submission
(12.5%)
StudyDesk
assignment
submission
(17.5%)
or

Page 4

Basnet et al., Peer assessment of assignments – USQ experience

Proceedings of the 2009 AaeE Conference, Adelaide, Copyright © B. Basnet, L. Brodie, and J. Worden 2009
items electronically on the StudyDesk while providing quality feedback to students via the completed
marking rubric.
Peer assessment and its evaluation
Student peer assessors assessed two randomly allocated assignments within the Turnitin system using
a marking rubric containing topic and metric questions. Each assignment question required answering
several topic questions and each assignment required answering one metric question. The responses to
topic questions provided descriptive comment about the quality of the answer while the response to
the metric question provided an overall rating of the assignment on a scale of 1 to 5. The instructor
evaluated peer assessment work on the basis of the responses provided to topic and metric questions
and awarded up to 12.5% marks allocated for this task.
Moderation process
Moderation involved two separate processes. Firstly, it involved comparing assignment marks
awarded by the instructor with those of the peer assessors’ ranking and reassessing those assignments
that were not concordant. This process resulted in a final mark for the first part of the assignment
carrying 17.5% marks. Secondly, it involved comparing peer assessment work with the alternate
assessment set for non-participating students. This process resulted in a final mark for the second part
of the assignment carrying 12.5% marks.
Data acquisition stage
Data addressing six research questions was acquired on completion of the assignment assessment
process. Prior to the data acquisition process, the medium necessary to acquire the data, the tasks to be
completed, and the research questions to be asked were identified as shown in Table 1 below. Data
were acquired using the three different methods: online quizzes, online survey, and discussion forum.
Table 1 Data acquisition mechanism
Research
Question
RQ 1
Students’
reaction
RQ 1
Students’
performance
Necessary medium to answer
the research question
Students’ discussions on peer
assessment system and their
responses to the survey question.
On-line quiz results. Discussions
on peer assessment system and
students’ responses to the survey
question.
Students’ discussions on peer
assessment system and their
responses to the survey question.
Students’ engagement in peer
discussions and their responses to
the survey question.
Tasks to be completed and a question to be
asked via survey questionnaire
Setup of discussion forum. Survey question:
What do you think about the peer assessment
system in general?
Conduct online quizzes. Setup of discussion
forum. Survey question: Has peer assessment
system helped to improve your understanding
of the course material?
Setup of discussion forum. Survey question:
Do you find peer assessment system a useful
learning tool in your learning journey?
Setup of ‘peer discussion’ forum. Survey
question: Did peer assessment system
instigate you to interact with fellow students?
RQ 2
Students’
experience
RQ 4
Students’
interaction
RQ5
Peers’
feedback
Student’s comments and concerns
about peer assessment via forum
discussions, e-mail, and responses
to the survey question.
Student’s suggestions via e-mail,
online discussion forum, and
online survey question.
Setup of discussion forum. Open e-mail
communication with students. Survey
question: What do you think about the
usefulness of the feedback from your peers?
Setup of discussion forum. Open e-mail
communication with students. Survey
question: What improvement (if any) to peer
assessment system do you suggest?
RQ6
Students;
suggestion

Page 5

Basnet et al., Peer assessment of assignments – USQ experience

Proceedings of the 2009 AaeE Conference, Adelaide, Copyright © B. Basnet, L. Brodie, and J. Worden 2009
Online quizzes
Two slightly different sets of online quizzes, based on 10 multiple-choice and 10 true or false type
questions, were developed from the course materials covered by the assignment. Quizzes differ
slightly to avoid a repetition effect on student performances. The first quiz was conducted soon after
the assignment submission, the second quiz after the completion of peer assessment. Students
completed these quizzes voluntarily. Class scores achieved in these quizzes by participating students
were analysed to infer differences in performance before and after the peer assessment.
Survey questionnaire
The student survey questionnaire was developed to address the research questions (see Table 1).
Likert-scale type survey questions were used. In a typical Likert-scale question, participating students
are asked to answer how much they agree with the issue of concern in a scale of one (strongly
disagree), two (disagree), three (not sure), four (agree) to five (strongly agree). Likert-scale based
questions were expanded to accommodate brief written explanations about their choice of answer. A
survey questionnaire was made available online soon after the completion of peer assessment. It
remains open until end of the semester.
Student discussion forum
An anonymous and open electronic discussion forum was setup for students’ discussion on peer
assessment. Students were encouraged to express their thoughts and feelings about peer assessment
without fear of any consequences. The discussion forum was made available before the
commencement of the peer assessment and it was left open until the end of the semester. Students
were reminded from time to time to contribute towards the discussion. The outcome of the discussion
forum was not intended to directly answer a particular research question but it was expected to provide
validation and valuable supplementary information.
Data analysis stage
The data analysis focused on two (i.e. numerical and text-based) categories of data. Analysis of
numerical data covered the assessment of student performances before and after the peer assessment
event. This analysis involved tracking the change in student performances between the two quizzes.
The numerical analyses also encompassed the assessment of the overall change in class performance
in this course as compared to the previous two years. Data analysis employed descriptive statistical
indicators such as mean and standard deviations as well as paired t-test to infer differences.
Survey data were divided into quantitative and qualitative data types for the purpose of analysis.
Quantitative data collected via Likert-scale type survey were analysed as ordinal data. Survey
responses were collated using bar charts. A central tendency and a weighted average were determined
and the dispersion was measured using quartiles.
Qualitative data is still being collected as part of this investigation and it includes; online survey
questionnaire, students’ discussions on peer assessment as well as students’ e-mail correspondence
with the instructor. Students’ discussions and e-mail feedback are to be summarised qualitatively. The
result is to be expressed in a tabular form to show total number of comments subdivided into
comments against, for, and neutral towards peer assessment. Suggestions and comments, added by the
students in the survey questions, discussion forum, and e-mail correspondence are expected to support
the answers to some of the research questions.
Preliminary results & discussions
Development of assessment items
The focus at the early stage of this research was to develop assessment items. Consequently, the
assignment based on seven composite assignment questions, clear marking guidelines, assignment
template and a comprehensive marking rubric (Appendix A) were developed. The basis for creating
these documents included such factors as; relevancy of the study materials in answering assignment
questions, clarity of instructions for the same purpose, a clear mark allocation guide and
confidentiality of the assignment for the purposes of peer review were all taken into account. Since,