Content uploaded by Kevin Ashford-Rowe
Author content
All content in this area was uploaded by Kevin Ashford-Rowe on Jan 01, 2019
Content may be subject to copyright.
University of Wollongong
Research Online
'#.*+#,0( (%%('!('!1+#+(%%,#('
'#.*+#,0( (%%('!('!1+#+(%%,#('+
A heuristic framework for the determination of the
critical elements in authentic assessment
Kevin Hugh Ashford-Rowe
University of Wollongong
+*"'%#'#+,"()'++#'+,#,-,#('%*)(+#,(*0 (*,"'#.*+#,0( (%%('!('!(* -*,"*#' (*&,#('(',,,"#**0
*+*")-+-(/--
(&&'#,,#('
+" (*(/.#'-!""-*#+,# *&/(*$ (*,",*&#',#('( ,"*#,#%%&',+#'-,"',#++++&',(,(*(
-,#(',"+#+-%,0( -,#(''#.*+#,0( (%%('!('!"2)*(-(/--,"++
A HEURISTIC FRAMEWORK FOR THE DETERMINATION OF
THE CRITICAL ELEMENTS IN AUTHENTIC ASSESSMENT
A thesis submitted in partial fulfilment of the requirements for the award of the
degree
DOCTOR OF EDUCATION
FROM
UNIVERSITY OF WOLLONGONG
BY
KEVIN HUGH ASHFORD-ROWE, BACHELOR OF ARTS
(HONOURS), POST GRADUATE CERTIFICATE IN EDUCATION,
GRADUATE DIPLOMA IN MULTIMEDIA, MASTER OF
PROFESSIONAL STUDIES, MASTER OF EDUCATION
FACULTY OF EDUCATION
2009
DECLARATION
I, Kevin H. Ashford-Rowe, declare that this thesis, submitted in partial fulfilment of the
requirements for the award of Doctor of Education, in the Faculty of Education,
University of Wollongong, is wholly my own work unless otherwise referenced or
acknowledged. The document has not been submitted for qualifications at any other
academic institution.
Kevin H. Ashford-Rowe
23 January 2009.
TABLE OF CONTENTS
Declaration ................................................................................................................ii
List of tables ..............................................................................................................vii
List of Figures.............................................................................................................vii
Abstract .............................................................................................................viii
Acknowledgments.......................................................................................................ix
Chapter 1: Introduction................................................................................................1
Background to the study — Authenticity in educational assessment..................... 1
Assessment, authenticity and educational technology............................................ 4
Research questions and the study ........................................................................... 8
The organisation of the thesis............................................................................... 10
Chapter 2: Authentic Assessment: A General review of the literature ................12
Previous findings .................................................................................................. 12
Assessment ........................................................................................................... 12
Assessment in higher education ........................................................................... 17
What is authentic assessment ............................................................................... 19
Assessment and educational technology .............................................................. 23
Characteristics of authentic assessment................................................................ 31
Chapter 3: Research Design and Methodology.......................................................37
Introduction .......................................................................................................... 37
Design-based research .......................................................................................... 37
PHASE 1: Exploration of the problem........................................................ 43
PHASE 2: Development of a solution ........................................................ 44
PHASE 3: Implementation and evaluation ................................................. 50
PHASE 4: Presentation of findings............................................................. 58
Summary of the research plan .............................................................................. 59
Conclusion ............................................................................................................ 62
Chapter 4: An Effective Model for Task Design in Flexible Learning
Environments......................................................................................63
Analysis of the elements of authentic assessment................................................ 63
Practitioner feedback ............................................................................................ 66
Evolving and further developing the critical elements ......................................... 69
1. Degree of challenge(s) presented to the assessed student....................... 70
2. Performance, or product, as final assessment outcome........................... 72
3. Transfer of learning (skills/knowledge/attitude) required....................... 74
4. Critical reflection and self-assessment or evaluation required................ 75
5. Accuracy in product or performance, and fidelity of assessment
environment, is displayed .................................................................. 76
6. Fidelity of assessment tools used ............................................................ 78
iii
Table of contents iv
7. Discussion and feedback required........................................................... 79
8. Collaboration required ............................................................................ 80
Expert review........................................................................................................ 81
Expert Reviewer 1....................................................................................... 82
Expert Reviewer 2....................................................................................... 85
Expert Reviewer 3....................................................................................... 88
Summary of feedback from expert reviewers....................................................... 90
1. Degree of challenge(s) presented to the assessed student....................... 91
2. Performance, or product, as final assessment outcome........................... 92
3. Transfer of learning (skills/knowledge/attitude) required....................... 92
4. Critical reflection and self-assessment or evaluation required................ 93
5. Accuracy in product or performance, and fidelity of assessment
environment, is displayed .................................................................. 93
6. Fidelity of assessment tools used ............................................................ 94
7. Discussion and feedback required........................................................... 94
8. Collaboration required ............................................................................ 95
Revision of critical elements from expert review................................................. 95
From critical elements to critical questions — A summary ................................. 98
The critical questions .................................................................................. 98
Chapter 5: Applying the Critical Questions of Authentic Assessment in
the Design of a Learning Module ....................................................100
Development of — Evaluating Educational Multimedia................................... 100
Introduction ............................................................................................... 100
The re-design of Evaluating Educational Multimedia ....................................... 101
1. To what extent does the assessment activity challenge the
assessed student? ............................................................................. 105
2. Is a performance, or product, required as a final assessment
outcome?.......................................................................................... 106
3. Does the assessment activity require that transfer of learning has
occurred, by means of demonstration of skill?................................ 107
4. Does the assessment activity require that metacognition is
demonstrated, by means of critical reflection, self-assessment
or evaluation?................................................................................... 108
5. Does the assessment require a product or performance that could
be recognised as authentic by a client or stakeholder?.................... 110
6. Is fidelity required in the assessment environment? And in the
assessment tools (actual or simulated)?........................................... 114
7. Does the assessment activity require discussion and feedback?.......... 115
8. Does the assessment activity require that students collaborate?.......... 116
9. Description of how the critical questions were applied in the
design and structure of the learning outcomes and assessment
criteria of Module 10 ....................................................................... 117
The role of formative assessment in the redesign of the module ....................... 133
The application of the elements to the learning environment ............................ 135
Conclusion .......................................................................................................... 136
Chapter 6: Learners’ Responses to Authentic Assessment ................................139
Learning Module Implementation...................................................................... 140
Learning Module Evaluation and Analysis ........................................................ 140
Table of contents v
Method of implementation........................................................................ 140
The method of analysis ............................................................................. 141
Applying the constant comparative method ....................................................... 142
Analysis of responses ......................................................................................... 143
Researcher’s observation on students’ responses by data source....................... 144
Interview ................................................................................................... 144
Observation ............................................................................................... 145
Video ......................................................................................................... 146
Notes on students’ performance made on observation during the delivery of
the module................................................................................................. 147
Notes on student’s performance made on researcher review of the video
content recorded during the delivery of the module ................................. 150
The student’s response to the critical questions ................................................. 152
1. To what extent does the assessment activity challenge the
assessed student? ............................................................................. 152
2. Is a performance, or product, required as a final assessment
outcome?.......................................................................................... 155
3. Does the assessment activity require that transfer of learning has
occurred, by means of demonstration of skill?................................ 157
4. Does the assessment activity require that metacognition, is
demonstrated, by means of critical reflection, self-assessment
or evaluation?................................................................................... 159
5. Does the assessment require a product or performance that could
be recognised as authentic by a client or stakeholder?.................... 160
6. Is fidelity required in the assessment environment? And the
assessment tools (actual or simulated)?........................................... 162
7. Does the assessment activity require discussion and feedback?.......... 163
8. Does the assessment activity require that students collaborate?.......... 165
Summary of the student’s response to the application of the critical
questions in the redesign of Module 10 .................................................... 167
The student’s response to the assessment activity.............................................. 170
Discussion........................................................................................................... 172
Chapter 7: Discussion .............................................................................................181
Research questions — Data analysis......................................................... 183
1. To what extent does the assessment activity challenge the
assessed student? ............................................................................. 186
2. Is a performance or product required as a final assessment
outcome?.......................................................................................... 188
3. Does the assessment activity require that transfer of learning has
occurred, by means of demonstration of skill?................................ 189
4. Does the assessment activity require that metacognition is
demonstrated by means of critical reflection, self-assessment
or evaluation?................................................................................... 190
5. Does the assessment require a product or performance that could
be recognised as authentic by a client or stakeholder?.................... 191
6. Is fidelity required in the assessment environment? And the
assessment tools (actual or simulated)?........................................... 193
7. Does the assessment activity require discussion and feedback?.......... 194
8. Does the assessment activity require that students collaborate?.......... 195
Table of contents vi
Summary of student response and impact on the critical questions.......... 199
Chapter 8: Conclusion.............................................................................................201
Introduction ........................................................................................................ 201
Summary and review of process......................................................................... 201
PHASE 1: Exploration of the problem...................................................... 204
PHASE 2: Development of a solution ...................................................... 204
PHASE 3: Implementation and evaluation ............................................... 206
PHASE 4: Presentation of findings........................................................... 207
Description of the principles............................................................................... 207
Findings of the study .......................................................................................... 209
Principal research question ....................................................................... 210
Subordinate research question 1................................................................ 211
Subordinate research question 2................................................................ 215
Conclusion .......................................................................................................... 217
Limitations of the study...................................................................................... 218
Recommendations for further research............................................................... 219
References ............................................................................................................221
APPENDIX 1 Expert Reviewer Interview Questionnaire ......................................239
APPENDIX 2 Student Evaluation Questionnaire ..................................................243
APPENDIX 3 Student interview Questionnaire.....................................................247
APPENDIX 4 Computer Based Learning Practitioners Course — Module
10 — Evaluating educational multimedia .......................................252
LIST OF TABLES
Table 3.1: The way in which the stages of the design-based research
process are applied in this study...........................................................42
Table 3.2: Summary of the research plan..............................................................61
Table 4.1: Researcher’s synthesis of the elements of authentic assessment
from the literature..................................................................................64
Table 4.2 Researcher’s translation of characteristics to critical elements of
authentic assessment with practitioner feedback .................................67
Table 4.3: Revision of critical elements from expert reviewer feedback to
produce the critical questions ...............................................................96
Table 5.1: Proposed application of the critical questions to the re-design of
Module 10 ...........................................................................................104
Table 6.1: Student Feedback on the Critical Elements........................................179
Table 7.1: Consideration of the students’responses with reference to the
research questions..............................................................................183
Table 7.2: Student Feedback on the Critical Questions.......................................198
Table 8.1: Stages of the design-based research process in this study................203
LIST OF FIGURES
Figure 3.1: Design-based research (2006, p. 59) ...................................................39
Figure 5.1: Apply the process of educational multimedia evaluation to the
Army’s Training Technology Centre developed Computer Based
Learning Practitioners Course ............................................................110
Figure 5.2: Apply the process of educational multimedia evaluation to a TTC
developed CBLP — Trainees will construct their own ........................113
Figure 5.3: Process of multimedia evaluation model assessment activity ............117
vii
ABSTRACT
Higher Education is currently undergoing a period of significant challenge and
transformation. It is likely that these challenges will, in a comparatively short period of
time, lead to changes in the ways in which the higher education experience is both
mediated and accessed. These changes have arisen as a result of a number of factors,
including the information revolution, and the consequent pace of technological
innovation, the increased demand from both employers and government for a more
highly skilled workforce and the desire to increase and make more accessible the higher
education experience to an increasing proportion of the overall population.
All of this has impacted upon the ways in which the higher education experience is
represented, and in turn, by which students gain access to the knowledge and skills that
will underpin their ability to both learn and perform. Higher education is increasingly
being challenged to demonstrate its continued value to the broader community,
especially employers, by ensuring that it provides capable, competent and informed
citizens adequate to the challenges of a twenty-first century lifetime. If these principles
are considered drivers for change, then it is important that the higher education sector
can continue to demonstrate its ongoing value to the students who undertake it.
It is against this background that this study was developed with the purpose of
identifying from the literature, and then to codify into an applicable framework, the
critical elements that would determine an assessment as being authentic. The study took
as its starting point the importance, in the current educational context, of being able to
determine the elements that define an educational experience as being an authentic one.
The research commenced with a review of the literature to identify and collate those
elements that had been identified by previous researchers in the field. Next these
elements, once refined iteratively in practice, were developed into a framework that
could be applied by the designer of instruction and assessment, in order to ascertain
whether such a framework could be used to support the design of a more authentic
assessment experience. This framework was then applied in practice and the student’s
response to the learning and assessment designed according to these elements was
evaluated, and the elements were further reviewed and revised upon the basis of this
data. Thus the study was conducted in four phases, in the first of which the researcher
explored the problem, in the second the researcher sought the development of a
solution, and in the third phase this solution was implemented and evaluated, the
findings were presented in the final phase.
The findings of this study suggest that not only is it possible to codify those elements
critical to the determination of authenticity into such a framework, but moreover, it is
possible to systematically apply them in the design of assessment activity. Thus the
implication of this research for educators and educational designers who seek to meet a
requirement for workplace relevance in the design of their education and assessment
activities is that they will have a better opportunity to both identify and then apply
specific design principles that will assist them in the better development of assessment
outcomes with a clearer workplace applicability.
viii
ACKNOWLEDGMENTS
In the completion of this work, I acknowledge the help, guidance and assistance freely
given to me by a number of people over many years, from both within and outside of
the field of education.
First, and foremost, I thank my wife, Tina, for her love and unstinting support, and my
children, Holly, Jamie and Sam, for their continued interest and the questions that have
ensured that I saw this through. I also thank all of them for the understanding that has
enabled me to sacrifice time with them to the completion of this study.
I thank my parents Ken and Catherine whose love, support, guidance and belief ensured
that both I, and my brothers, Jeremy, Alan and Ian, have been adequate to the joys and
challenges that a life can present.
From an educational perspective, I acknowledge the inspiration and assistance given to
me by many people, in particular, Robert Pepper and Lotte Deeble, Bosvigo County
Primary, Cornwall (1971–1974), Rosemary King, Penweathers County Secondary and
Richard Lander Comprehensive, Cornwall (1974–1979) and Margaret Garland and
David Worley of Cornwall College, Cornwall (1979–1982).
I also sincerely acknowledge the assistance and advice of Professor John Hedberg and
Professor Barry Harper.
Finally, I am deeply indebted to, and thank, my supervisors Associate Professor Jan
Herrington and Doctor Christine Brown, who have provided me with both support and
guidance over the last several years, and without whose vision and the means to express
it, I would not have come close to completing this work.
I dedicate this work to my father, Leslie Kenneth Rowe, Kernowyon, (1931–1992) who
understood the true value and importance of education as the enabler that can allow us
to fulfil our potential and raise ourselves up.
My thanks to Jill Ryan for her excellent assistance in proof reading this thesis.
ix
‘Then said a teacher, Speak to us of Teaching.
And he said:
No man can reveal to you aught but that which already lies half asleep in the dawning
of your knowledge.
The teacher who walks in the shadow of the temple, among his followers, gives not of
his wisdom but rather of his faith and his lovingness.
If he is indeed wise he does not bid you to enter the house of his wisdom, but rather
leads you to the threshold of your own mind.’ (Khalil Gibran, 1923, p. 74)
x
