Content uploaded by Linda Candy
Author content
All content in this area was uploaded by Linda Candy
Content may be subject to copyright.
Practice Based Research: A Guide
Linda Candy
Creativity & Cognition Studios
http://www.creativityandcognition.com
University of Technology, Sydney
CCS Report: 2006-V1.0 November
Summary
This document characterises practice-related research for the general reader and research student.
There are two types of practice related research: practice-based and practice-led:
1. If a creative artefact is the basis of the contribution to knowledge, the research is
practice-based.
2. If the research leads primarily to new understandings about practice, it is practice-led.
Practice-based Research is an original investigation undertaken in order to gain new
knowledge partly by means of practice and the outcomes of that practice. In a doctoral thesis,
claims of originality and contribution to knowledge may be demonstrated through creative
outcomes in the form of designs, music, digital media, performances and exhibitions. Whilst
the significance and context of the claims are described in words, a full understanding can
only be obtained with direct reference to the outcomes.
Practice-led Research is concerned with the nature of practice and leads to new knowledge
that has operational significance for that practice. In a doctoral thesis, the results of practice-
led research may be fully described in text form without the inclusion of a creative work. The
primary focus of the research is to advance knowledge about practice, or to advance
knowledge within practice. Such research includes practice as an integral part of its method
and often falls within the general area of action research.
The primary focus of this document is on practice-based research but there is much that is
relevant to practice-led research also. It begins with a discussion of the basic concepts in the
context of a doctoral research programme followed by a brief historical overview of the field.
The generic structure of a practice-based doctoral thesis is then outlined with a short
description of the expected content of each chapter. Further sections include a discussion of
the nature of knowledge in the context of doctoral research, a set of frequently asked
questions, some definitions of key terms, a bibliography and web sources.
Document Overview
• Practice and Research
• Defining Practice-based Research
• Historical Background
• The PhD and Knowledge
• Thesis Outline
• Questions and Answers
• Bibliography
• Definitions and Terms
© Linda Candy CCS
Thank you to CCS researchers: Ernest Edmonds, Zafer Bilda, Brigid Costello, Mike Leggett, Sarah Moss, Lizzie Muller, Julien
Phalip, Greg Turner, Yun Zhang
2
Practice and Research
Research that takes the nature of practice as its central focus is called ‘practice-based’ or
‘practice-led’ research. It is carried out by practitioners, such as artists, designers, curators,
writers, musicians, teachers and others, often, but not necessarily, within doctoral research
programmes. This kind of research has given rise to new concepts and methods in the
generation of original knowledge.
It is important to make a clear distinction between practice-based research and pure practice.
Many practitioners would say they do ‘research’ as a necessary part of their everyday practice.
As the published records of the creative practitioners demonstrate, searching for new
understandings and seeking out new techniques for realising ideas is a substantial part of
everyday practice. However, this kind of research is, for the most part, directed towards the
individual’s particular goals of the time rather than seeking to add to our shared store of
knowledge in a more general sense. Scrivener argues that the critical difference is that
practice-based research aims to generate culturally novel apprehensions that are not just novel
to the creator or individual observers of an artefact; and it is this that distinguishes the
researcher from the practitioner (Scrivener, 2002).
Another important distinction between personal practitioner research and doctoral practice-
based research is the form that the knowledge generated takes. The practice-based doctoral
research outcome that is shared with a wider community arises from a structured process that
is defined in university examination regulations.
In order to achieve advances in knowledge of the kind referred to above, the everyday
research process common to professional practice has to be defined and executed in a manner
that is commonly agreed. The research component of the practice-based research is, in most
respects, similar to any definition of research, a key element of which is the transferability of
the understandings reached as a result of the research process.
In the UK, the Arts and Humanities Research Board (now Council) (AHRB, 2000) defined
research primarily in terms of research processes rather than outputs. This definition is built
around three key features of any doctoral research proposal:
1. It must define a series of research questions or problems that will be
addressed in the course of the research. It must also define its objectives in
terms of seeking to enhance knowledge and understanding relating to the
questions or problems to be addressed.
2. It must specify a research context for the questions or problems to be
addressed. It must specify why it is important that these particular questions
or problems should be addressed, what other research is being or has been
conducted in this area and what particular contribution this project will
make to the advancement of creativity, insights, knowledge and
understanding in this area.
3. It must specify the research methods for addressing and answering the
research questions or problems. In the course of the research project, how to
seek to answer the questions, or advance available knowledge and
understanding of the problems must be shown. It should also explain the
rationale for the chosen research methods and why they provide the most
appropriate means by which to answer the research questions.
Creative output can be produced, or practice undertaken, as an integral part of the research
process. However, the outcomes of practice must be accompanied by documentation of the
research process, as well as some form of textual analysis or explanation to support its
position and to demonstrate critical reflection. A thesis arising from a practice-based research
process, such as the one given above, is expected to both show evidence of original
scholarship and to contain material that can be published or exhibited.
3
Practice-Based and Practice-Led Research
Although practice-based research has become widespread, it has yet to be characterised in a
way that has become agreed across the various fields of research where it is in use. To
complicate matters further, the terms ‘practice-based’ and ‘practice-led’ are often used
interchangeably. In reality, there are two main types of research that have a central practice
element and that distinction is summarised here as follows:
• If a creative artefact is the basis of the contribution to knowledge, the research is
practice-based.
• If the research leads primarily to new understandings about practice, it is practice-led.
Practice-Based Research
Practice-based Research is an original investigation undertaken in order to gain new
knowledge partly by means of practice and the outcomes of that practice. Claims of originality
and contribution to knowledge may be demonstrated through creative outcomes which may
include artefacts such as images, music, designs, models, digital media or other outcomes such
as performances and exhibitions Whilst the significance and context of the claims are
described in words, a full understanding can only be obtained with direct reference to those
outcomes. A practice-based PhD is distinguishable from a conventional PhD because creative
outcomes from the research process may be included in the submission for examination and
the claim for an original contribution to the field are held to be demonstrated through the
original creative work.
Practice-based doctoral submissions must include a substantial contextualisation of the
creative work. This critical appraisal or analysis not only clarifies the basis of the claim for the
originality and location of the original work, it also provides the basis for a judgement as to
whether general scholarly requirements are met. This could be defined as judgement of the
submission as a contribution to knowledge in the field, showing doctoral level powers of
analysis and mastery of existing contextual knowledge, in a form that is accessible to and
auditable by knowledgeable peers.
Practice Led Research
Practice-led Research is concerned with the nature of practice and leads to new knowledge
that has operational significance for that practice. The main focus of the research is to advance
knowledge about practice, or to advance knowledge within practice. In a doctoral thesis, the
results of practice-led research may be fully described in text form without the inclusion of a
creative outcome. The primary focus of the research is to advance knowledge about practice,
or to advance knowledge within practice. Such research includes practice as an integral part of
its method and often falls within the general area of action research. The doctoral theses that
emerge from this type of practice related research are not the same as those that include
artefacts and works as part of the submission.
Domain Differences in Practice-based and Practice-led Research
The use of the term practice-based research has become widespread but it has yet to be
characterised in detail in a way that is agreed across the various fields of research where it is
in use. There are differences in conceptual and applied uses of the term between those fields
where it is most often found: design, health, creative arts, and education. There are, in fact,
differences in the type of research in respect of whether it is practice-based or led.
In design research, for example, where the nature of practice is a major research topic and is
often conducted by research specialists rather than design practitioners, the emphasis is on
achieving new knowledge about the nature of practice and how to improve it, rather than
creating and reflecting on new artefacts. By contrast, in the visual arts, the emphasis is on
creative process and the works that are generated from that process: the artefact plays a vital
part in the new understandings about practice that arise.
It is important to note that the term practice-based research rather than practice-led research
is used the area of health research. Here it may include any of the following, but not limited
to: literature reviews including systematic examination of intervention and outcome measures,
program evaluation clinical trials, evaluation or revision of health care protocols, policies and
procedures, pilot projects and peer-reviewed studies.
4
Historical Background
Practice-based PhDs began in Australia in 1984, when the University of Wollongong and the
University of Technology, Sydney (UTS) introduced Doctorates in Creative Writing. Graeme
Harper obtained the first such degree in Australia from UTS. He is currently at the University
of Plymouth in England, where he is very active in promoting practice-based research. Two
current UTS professors, Theo van Leeuwen and Ernest Edmonds, were active earlier in such
UK developments.
The first Polytechnic was formed in London in1880. When the concept was used for the
significant expansion of such institutions in 1968, the goal was to add a service element to the
mainstream of higher education. The knowledge that the Polytechnics taught and developed
was to emphasis value in practice. Higher education was no longer to be seen as the centre of
new understanding, of knowledge that described the world, but as the centre of new ways of
doing things, of knowledge that improved our ability to act in the world.
When the Council for National Academic Awards (CNAA) drew up its regulations for the
higher degrees to be awarded from Polytechnics, they included a critical clause, “The written
thesis may be supplemented by material in other than written form”. This enabled a student to
include an artefact, or the record of an artefact, as an integral part of their PhD submission.
For example, when Susan Tebby submitted her PhD, “Patterns of Organisation in Constructed
Art”, (Tebby, 1983), she put up an exhibition and included a full set of 35mm slides of its
contents bound with the thesis. The examination was based on the artworks and the written
thesis together. Practice-based PhDs today are most simply identified by the inclusion of such
artefacts within the submission.
In Australia, the Australian Research Council has been funding research in creative practice
and has entered a partnership with the Australian Council in which collaborative art/science
projects are funded jointly. Similarly, in the UK, the Arts and Humanities Research Council,
in its definition of research, states that “Creative output can be produced, or practice
undertaken, as an integral part of a research process”.
The crucial point is that in certain disciplines knowledge can be partly advanced by means of
practice. The idea that has developed was that a research student, for example, would take, as
the subject of research the practice of their own discipline. The research programme would
consist of a continual reflection upon that practice and on the resulting informing of practice.
The examination would be based upon both the results of the practice and on a thesis
concerning the reflections upon the process undertaken. Thus artefacts, for example a set of
objects that had been designed and constructed, would form part of the candidate’s submission
for the degree. The candidate would be expected to satisfy the examiners in all of the ways
that are normal in a doctorate, such as demonstrating that they are well acquainted with the
general field of knowledge in which their subject relates. It would be required that the thesis,
as lodged in the Library, would include a permanent record of any artefacts submitted towards
the examination. In this way, the practice-based PhD can be understood within the traditional
context of the purely written PhD without any major revolution in education being required.
Studies have been made of practice-based research supervision that help to illuminate the
process. For example, in order to chart the experiences of such students, qualitative interviews
were undertaken with 50 research students at various UK universities. A paper based on those
interviews examines one dimension of how students adapt to this kind of study, focusing on
their conceptions of identity (Hockey, 2003)
5
The PhD and Knowledge
The PhD is defined in terms of knowledge, e.g. “The degree is awarded to candidates who,
through original investigation, make a distinct and significant contribution to knowledge.”
(UTS, 2006). That new knowledge is expected to have two characteristics: it is shared and it
can be verified or challenged. So a PhD describes knowledge that is new (in the world), can be
shared with others and can be tested in some way. Accepting that much of what we know is
known tentatively rather than absolutely, the properties of being sharable and challengeable
are more important than the absolute certain truth of the new knowledge.
Preliminaries
There are a few knowledge issues that are worth mentioning but are not, in fact, relevant in
our context. In each case, of-course, the issues could be the subject of a PhD in Philosophy
itself.
• Tacit knowledge
That we have achieved new implicit or tacit knowledge is clearly not relevant because, by
definition, it is not shared.
• Subjective or private knowing
Any position that argues that knowledge is private to an individual argues that knowledge
cannot be shared and so rules against the possibility of research in the PhD sense.
• Pure argumentation
The new knowledge can be that B follows from A. In other words, it is shown that, presuming
(but not claiming) that A is true, then B must be true also. Such an argument does not appeal
to any facts about the world but relies entirely on the strength of its argument. This is not a
matter of rhetoric, it has to be a matter of logic. In mathematics, for example, such PhDs are
common. The crucial issue is to show that the system of reasoning used and the assumptions
A do not contain or imply a contradiction. That can be quite hard. Reaching into the
philosophical foundations, however, is a serious matter, not least because of Gödel’s theorem
(Gödel, 1931). In any case, a PhD based upon pure argumentation cannot be practice-based.
Knowledge
A classic text is A. J. Ayer’s “The Problem of Knowledge”. He argues, pragmatically, that we
needed to find a “right to be sure” to support a belief in order to call it knowledge
(Ayer,1956). It is a matter of debate how strong the backing needs to be for a believer to have
the right to be sure that their belief is true. However, The setting of a standard requiring the
impossibility of error should be resisted. His view was that one has the right to “be sure” even
where error is possible.
A huge influence has been Karl Popper’s view of a “right to be sure” expressed in his
description of scientific enquiry (Popper, 1959). His key point is that we cannot know any
general truth about the world for sure. Many, Bertrand Russell for example Russell, 1912),
have pointed out that we can only observe a finite number of events and that, for all we know,
the next observation will contradict any theory we have based on the earlier ones. Thus Popper
argued that the pursuit of new knowledge was based on attempting to falsify our current
hypotheses or beliefs. The longer we go on failing to falsify them the more we can claim the
right to be sure about them. Once falsified, we need to find a new or modified theory. The
main thing is to be open to, and even invite, criticism and attempts to disprove our theories.
Much of the understanding about having knowledge rests upon what it is that we perceive in
the world and that others also report perceiving. The problem is that, of-course, we have direct
knowledge of what are often called “sense data” (Swartz, 1965) but only indirect knowledge
of the world through those data. A significant stance in this context is constructivism (not to
be confused with movements of the same name in art and in mathematics) (Bruner, 1986). In
this view, in our pursuit of knowledge about the world we construct it rather than uncover it.
One might argue that “being sure” is about being sure that it is plausible rather than that it is
true.
Beyond knowing what is we are also interested in what causes or influences what is. David
Hume’s work is classic here (Hume, 1777). He articulated the “problem of induction”.
Basically, we can never be sure that X causes Y because we cannot reliably induce the general
case from specific instances (as above). However, he gave a valuable lesson in how to deal
with such philosophical problems. He said that we had to rely on such induction in ordinary
life or we would “perish and go to ruin.” So some philosophical problems about knowledge
6
may be intractable or devastating, but we must carry on anyway. After all, despite the problem
of induction, for example, we can boil a kettle, send an email and ride a bicycle.
Beyond knowing what is and knowing what causes what, we are also interested in knowledge
about action. We are clearly able to find new knowledge about how to better achieve some
end. Practical knowledge of this kind can still be shared, verified and criticised. ‘Knowledge
how’ may not, however, provide the degree of explanation that ‘knowledge that’ does. The
problem of induction is less of a concern. The action researcher might generate new
knowledge about how to do something but leave it open to others to discover why it works. A
phenomenologist might argue that this ‘knowing how’ precedes ‘knowing that’. From that
point of view, action research should come before experimental research. Until the action
research is complete, it could be argued, we do not know what to study experimentally. If we
were starting from a clean sheet of zero knowledge, perhaps that would be true, but reality is
more complex.
Phenomenology (Lewis, 1946) has a number of strands, but one important concept is that, to
put it very briefly, the body is important in perception. In practical terms, this implies that the
typical theories of cognitive science may be fatally lacking. A reliable account of perception
must take action into account. Further, one can argue that action, cognition and perception
must be considered together in any adequate description. If we accept this view, then research
about human interaction with art works, for example, must try to capture information about all
three aspects and unify them in some way.
Frederick Crews put the point about how to approach knowledge nicely. He said that we
should follow “the ethic of respecting that which is known, acknowledging what is still
unknown and acting as if one cared about the difference” (Crews, 2006). Finally, a lighter read
on writing about these topics is Francis Wheen’s “How Mumbo-Jumbo Conquered the
World” (Wheen, 2004). Chapter four, called “The Demolition Merchants of Reality” should
be required reading for any PhD candidate.
References
Arts and Humanities Research Board (2000). Guide to the Research Grant Scheme.
http://www.ahrb.ac.uk/research/grant/guide.htm [accessed July 2000]
Ayer, A.J. (1956). The Problem of Knowledge, Macmillan Press, London.
Bru ner, J. (1986 ). A ct ual M i nds, P o ssi b le World s. Cambri dge, MA: H arvard U n iver sity P ress.
Crews, F. (2006). Follies of the Wise, Shoemaker and Hoard. Emeryv ille, CA.
Gödel, K. (1931). Über formal unentscheidbare Sätze der Principia Mathematica und verwandter
Systeme, I. Monatshefte für Mathematik und Physik 38: 17 3-98 . Translated in Jean van
Heijenoort, 1967. From Frege to Gödel: A Source Book on Mathematical Logic. Harvard
University Press: pp 596-616
Gödel’s theo rem: http://en.wik iped ia.org/wiki/Gödel's_incompleteness_theore
Hockey, J. Art and Design Practice-Based Research Degree Supervision. Arts & Humanities in
Higher Education. vol 2 (2) 173–185.
Hume, D. (1777) An Enquiry concerning Human Understanding. Beauchamp, T. L. (ed.), (1999),
Oxford University Press, Oxford.
Hume, D. ttp://en.wikipedia.org/wiki/David_Hume
Lewis, C. I. (1946) An Analysis of Knowledge and Valuation. Open Court, USA.
Popper, K. R . (1959 – English translation) The Logic of Scientific Discovery (2002) Routledge,
London.
Russell, B., A., W. (191 2) The Problems of Philosophy, Williams and Norgate, London
Scrivener, S.A.R (2002). The Art Object Does Not E mbody a Form of Knowledge, Working Papers
in Art and Design, http://www .herts.ac.uk/artdes1/research/pap ers/wp ades/vol2/scrivenerfull.htm
Stringer, E.T (2003) Ac ti on Res ea rch in E du cat ion , Prentice Hal
Swartz, R., J. (ed.) (1965) Perceiving, Sensing and Knowing. Doubleday, New York.
UTS (2006). Doctor of Philosophy, University of Technology, Sydney.
http://www.h andbook.uts.edu.au/it/pg/c02029.html
Wheen, F (2004). How Mumbo-Jumbo Conquered the World, Harper, Lon don.
7
Thesis Outline
1. Introduction
Four key elements are briefly described in this section.
i.The Problem
This is a concise statement of the research question or issue that the thesis addresses.
ii.The Context
What is the main work that has been done that gives rise to the question and what is its
significance?
iii.The Method
The approach to solving the problem (experimental, practice based, analytic etc) is described
in this part, leaving the justification to chapter three.
iv.The Outcomes
Here the key contribution(s) to knowledge are concisely described. They are the things that
arise from the work that are new and shown to advance understanding or practice
internationally. The value of these outcomes will be to one or more community (computer
scientists, artists, theoreticians etc) and it is important to be clear who they are.
2. State of the Art Review
This chapter presents the results of a literature survey of the area(s) of study. It should be a critical
review in the context of the stated research question and related issues. This chapter answers questions
such as: Who is doing what? Who has done what? Who first did it or published it? The survey is taken
from published papers, research monographs, catalogues etc. It must be based on and refer to primary
sources, not textbooks or other such reports on the work of others. It is to be expected that this chapter
provides a new structured view of the field of study.
3. Methodology
This is a key chapter that provides a description and justification of the research methods used.
Normally, the methods will be selected from known and proven examples. In special cases the
development of a method may be a key part of the research, but then this will have been described in
section one and reviewed in two.
4. Foundation Work
This, optional, chapter is a chance to describe earlier work done by the author/candidate (possibly with
others) that provides a foundation or significant background. It may be helpful to revisit and reassess
earlier work in the light of the research focus of the PhD. This chapter will not be needed in some PhDs
where the work is from a fresh start.
5. New Studies
The core of the thesis is a description of the new studies/software/artwork and the process of production.
It answers the questions: What has been done, how was it achieved and what was the rationale? This can
be, for example, a report on the design and execution of a set of experiments or the development of an
innovative software system or the making of innovative art works. In a practice based PhD an artwork,
for example, might be presented for examination. If so, this chapter will illuminate it by explaining, at
the very least, what is important and novel
6. Results
The evaluation of the new software/artwork or analysis of the results or processes of the new studies will
have led to certain results or conclusions. Placing the new results in the context of Chapter 4 is
important. The outcomes, as promised in chapter 1, are shown to have been achieved in this section.
7. Conclusions
A discussion can now be provided that puts a wider perspective on the results and discusses the
implications of them for other broader areas and domains. Future work and outstanding questions are
normally also discussed.
8. References/Bibliography (including published papers)
Use a standard reference format, such as Harvard, and be careful to check each entry. It is temping to
presume that software such as End-Notes will ensure a perfect reference list, but that all depends on
exactly how each entry was stored. There is no substitute for a line-by-line check.
8
Questions and Answers
Q1. What is the purpose of the Literature Review?
A1. The Literature Review should cover that work by others which provides a basis and
context for your research either because you are using the findings to support an argument to
extend something or because you have identified a limitation and will be pursuing studies to
address it. The review should be drawn from primary sources: e.g. papers reporting results of
original research. Do not quote someone’s reference to someone else’s work: always go to the
original text.
Q2. Should I place the background discussion of the origins
and source of my methodology in the methodology chapter
or in the Literature Review?
A2. Normally, you would describe the origins of your methodology in the methodology
chapter. Where the contribution to knowledge is primarily a methodological one it might be
appropriate to describe the state of the art in the Lit Review chapter and have a short summary
of the approach in the methodology chapter. But this is exceptional.
Q3. How much detail should I go into when describing the
methodological approach?
A3. Some tips are:
1. start with the actual things you have done/will do (as if instructing an assistant)
2. then add a brief description of the origins of the method(s) - sources from which
they have been selected
3. if any aspect is not yet justified develop the argument to justify it.
Number 3 should be minimal. If the argument includes negatives ("statistics is no good for
this" or "semiotics does not work in this case") be sure that you are knowledgeable about what
you are rejecting.
Q4. How should I write reflective documentation?
A3. Monitoring and Recording ‘Events’ for Reflective Practice
• Keep a written record either in the form of an online diary or blog, or a hand written
notebook.
• Having a notebook with you at all times to record any observations or events during
the process of making a work is advisable.
• If you are collecting audio records, you will probably need to transcribe them at some
point and may need to factor that in to your time and effort schedule.
• Keep an overview chart to be able to see at a glance what has been recorded and what
you plan to record.
• It is advisable to designate time for reflection after the events and to record your
reactions and emerging thoughts at specified Review points. Do not be tempted to
only review and reflect on an ad hoc basis. Build it in to your timetable.
Good questions to ask yourself are:
- what was proposed, discussed, decided and carried through,
- what stumbling blocks arose and how they were addressed….,
- whether the ideas were workable, interesting, challenging….
- whether the collaboration worked well or not
- reasons for success or otherwise
- did the solutions work well, if not why not?
- whether there were different viewpoints between you and your collaborating parties
- whether lessons were learnt from failures.
9
Q5. What is the role of the artefact in reporting the results?
A5. The artefact is not an explanation in itself:
- it requires linguistic description that relates the development and nature of the
artefact to understandings about creative process
- the text describes the innovation embodied in the artefact but cannot be fully
understood without reference to and observation of the artefact.
Viewpoints
Ross Gibson’s view is 'the text is not an explanation of the artwork; rather, the text is an
explicit, word-specific representation of processes that occur during the iterative art-making
routine, processes of gradual, cyclical speculation, realisation or revelation leading to
momentary, contingent degrees of understanding. To this extent the text that one produces is
a kind of narrative about the flux of perception-cognition-intuition. The text accounts for the
iterative process that carries on until the artist decrees that the artwork is complete and
available for critique, 'appreciation', interpretation, description, evaluation. All these particular
practices can entail other particular texts.'
Steve Scrivener’s view: ‘The art object does not embody a form of knowledge’
o Art is not a form of knowledge communication
o Art is not a servant of knowledge acquisition
o Art making creates apprehensions
o Art research creates novel apprehensions
See http://www.herts.ac.uk/artdes/research/papers/wpades/vol2/scrivener.html
Ethical Considerations
All research that involves people in the collection and analysis of data is subject to ethical
considerations. The first stage in designing a research project is to prepare a research proposal
which outlines the proposed methodology and any ethical issues. Researchers must obtain the
informed consent of persons participating in research before the research begins. Researchers
should provide participants with accurate information about the purpose, methods, demands,
risks, inconveniences and discomforts of the study. Information should be at a level
appropriate for comprehension by research subjects. Each institution in which a researcher
works will have an Ethics Approval procedure which must be followed prior to undertaking
the research. Examples from CCS University of Technology, Sydney are available.
See CCS Code of Ethics document
See CCS Procedures and Application Form
10
Bibliography
Action Research
Action Research Electronic Reader
http://www.scu.edu.au/schools/gcm/ar/arr/arow/default.html
Stringer, E.T (2003) Action Research i n Education, Prentice Hall
Argyris, C. R. Puttnam, R and McLain Smith, D. Action Science.
http://www.actiondesign.com/action_science/
Data Collection and Analysis Methods
Richards, L. (2005, Handling Qualitative Data, Sage, London.
Bilda, Z., Costello, B. & Amitani, S. (2006) Collaborative Analysis Framework for Evaluating
Interactive Art Experience, Co-Design Special issue.
Suwa, M., Purcell, T. & Gero, J. (1998). Macroscopic Analysis of Design Processes Based on
a Scheme for Coding Designers' Cognitive Actions, Design Studies, vol. 19, no. 4, pp. 455-
483.
Discourse analysis
Hammersley, M. (2002), Discourse analysis: A Bibliographical Guide
http://www.cf.ac.uk/socsi/capacity/Activities/Themes/In-depth/guide.pdf
MacMillan, K. Discourse Analysis — A Primer
http://www.lboro.ac.uk/research/mmethods/resources/links/da_primer.html
Potter, J. (1996), “Discourse Analysis and Constructionist Approaches: Theoretical
Background" In: John T.E. Richardson (ed.): " Handbook of qualitative research methods for
psychology and the social sciences," Leicester: BPS Books.
Conversation analysis
Antaki, C. (2005) An Introduction to Conversation Analysis
Available on line at: http://www-staff.lboro.ac.uk/~ssca1/intro1.htm
Psathas, G (1994) Conversation Analysis : The Study of Talk-in-Interaction Sage, Newbury
Park, London
Schegloff, E.A.
Available on line at: http://www.sscnet.ucla.edu/soc/faculty/schegloff/
Survey
On-line resource about how to conduct survey
http://www.managementhelp.org/commskls/surveys/surveys.htm
Interviews
http://www.ul.ie/~infopolis/methods/interv.html
Brief information sheets from University of Illinois Extension on ‘how to do interview’
http://www.aces.uiuc.edu/~PPA/KeyInform.htm
Bob Dick. http://www.scu.edu.au/schools/gcm/ar/arp/iview.html A technique for qualitative
data collection. How to do interviewing.
Introduction to Interviewing Techniques
http://www.wpi.edu/Academics/Depts/IGSD/IQPHbook/ch11.html
11
Kvale, S. (1996) InterViews: An Introduction to Qualitative Research Interviewing
Steinar Kvale, Institute of Psychology, University of Aarhus, Denmark
This is an introduction to interviewing outlines both the theoretical underpinnings and the
practical aspects of the process. After examining the role of the interview in the research
process, Steinar Kvale considers some of the key philosophical issues relating to interviewing:
the interview as conversation, hermeneutics, phenomenology, concerns about ethics as well as
validity, and postmodernism. Having established this framework, the author then analyzes the
seven stages of the interview process - from designing a study to writing it up.
Observation
Data collection methods:
http://www.acf.dhhs.gov/programs/ccf/resources/gbk_om/om_gbk_dcm.html
J Lofland, LH Lofland (1995). Analyzing Social Settings: A Guide to Qualitative Observation
and Analysis”, Wadsworth
Protocol analysis
Ericsson, K.A. & Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data, Cambridge,
MA: MIT Press.
Ericsson, K., & Simon, H. (1987).Verbal reports on thinking, In C. Faerch & G. Kasper (eds.)
Introspection in Second Language Research. Clevedon, Avon: Multilingual Matters, pp24–54.
Video-cued Recall
Costello, B., Muller, L., Amitani, S. & Edmonds, E. 2005, 'Understanding the experience of
interactive art: Iamascope in Beta_space', Proceedings of the second Australasian conference
on Interactive entertainment, Sydney, Australia pp. 49 - 56
Ericsson, A. and Simon, H. Protocol Analysis: Verbal Reports as Data. MIT Press,
Cambridge, MA, 1993.
Omodei, M., Wearing, A.J. and McLennan, Head-Mounted Video and Cued Recall: A
Minimally Reactive Methodology for Understanding, Detecting and Preventing Error in the
Control of Complex Systems. in 21st European Annual Conference of Human Decision
Making and Control,(Glasgow, 2002), Department of Computer Science, University of
Glasgow, Scotland.
Video Data Analysis
Candy, L, Bilda, Z., Maher, M-L. & Gero, J. (2004). Evaluating Software Support for Video
Data Capture and Analysis in Collaborative Design Studies. Proceedings First International
Conference on Qualitative Research in IT & IT in Qualitative Research, Brisbane, November
24-26.CD published by the Institute for Integrated and Intelligent Systems, Griffith
University: http://iiis.griffith.edu.au
INTERACT User Guide (2004) Mangold Software and Consulting GmbH.
Suchman L. and Trigg R. (1991) Understanding practice: Video as a medium for reflection
and design, in J. Greenbaum and M. Kyng (eds) Design at Work: Cooperative Design of
Computer Systems, Erlbaum, Hillsdale, N.J.: 65-90.
Drawing process analysis
Digital Sketching Website (in progress) http://www.uoregon.edu/~arch/digsketch/
12
Software for Data Analysis
The need for handling large qualities of observational data has given rise to software
applications that allow the researcher to collate the audio, video, text material and perform
various forms of qualitative and quantitative analysis.
Computer aided qualitative data analysis packages such as NVivo 2.0 do not directly support
the analysis of video. Other tools that do support video, such as Atlas/ti, Qualrus and Hyper
Research, are designed to work with small edited video clips, not with large libraries of raw or
unedited videos. Transana handles videos more effectively than the mentioned ones and it is
largely used because it is freeware, on the other hand Observer and Interact are higher end,
more professional software which offer more flexibility for data collection and analysis across
many disciplines.
NVivo where data are stored as sets of text documents, which can include field notes,
interview transcripts, communications between individuals (especially email
communications), and any other form of textual data. It is a text based analysis tool, which
does not provide us with the synchronous analysis of audio video data. (www.qsr.com.au).
Atlas.ti is an application for the visual qualitative analysis of large bodies of textual, graphical
and audio video data. The software supports all areas of document based research and works
with video, image, audio data attached to text. (http://www.atlasti.de/)
"HyperRESEARCH" which enables the coding of any type of source including text, audio,
video and image. HyperRESEARCH™ is a qualitative data analysis software package
enabling users to code and retrieve, build theories, and conduct analyses of observational data.
Recently with advanced multimedia capabilities, HyperRESEARCH allows users to work
with text, graphics, audio, and video sources. The video recordings or captured images could
be attached to text records to support heterogenous analysis (more information can be found at
http://www.researchware.com/).
Transana is a free software for analyzing video and audio data. User can organize video clips
(from the same or from different video files) into meaningful categories, as a mechanism for
developing and expanding the theoretical understanding of what the video shows. The
software allows one to apply searchable analytic keywords to these video clips. Search tools
allow data mining and hypothesis testing across large video collections. It is also possible to
share analytic markup with distant colleagues to facilitate collaborative analysis (more
information can be found on www.transana.org ).
Observer (by Noldus) is for the collection, analysis, presentation and management of
observational data. Noldus, providing a wide range of features for instrumentation for data
collection and analysis, their software Observer can be applied to study observable behavior
such as activities, postures, gestures, facial expressions, movements, and social or human-
system interactions. The latest version of Observer integrates analysis of textual data and
handling transcripts to the video data. Observer also integrates different data collection
techniques with accessories to suit different research environments (field/ lab/ dynamic) such
as event logging interface, eye trackers, mobile device camera, spectacles camera, and so on.
Specific data collection and analysis tools vary according to the needs of the discipline
(Neuroscience, Psychology, Zoology, Usability) and Observer XT provides with add-in tools
to satisfy these needs. U-Log, logging tool, automatically records user computer interaction,
EthoVision, a video tracking system is used for automatic recording of activity, movement
and interactions of animals; and Theme is used for detection and analysis of patterns in time-
based analysis. More information can be found at http://www.noldus.com.
INTERACT is a software tool that helps you to save hundreds of hours of time during analysis
of video recordings and live observations, offering sophisticated logging and analysis
methods. Interact can be used to study observable behavior such as activities, postures,
gestures, facial expressions, movements, and social or human-system interactions. With
Interact it is possible to record events (segments of information based on video) independent
of a method. Events can be logged simultaneously in one video, and the ways to achieve it
very flexible. Interact allows annotation, integration of text/ transcripts connected to the
videos events, and it can handle any type of multimedia file as well as the functionality to
control video devices directly. Interact has add-ins for specific research disciplines (Neuro
Science, Psychology, animal behavior, Usability). With Data view user can display any kind
13
of externally acquired data synchronously to the video recordings, such as physiological
information, or data acquired by mechanical systems. With Sound Analyzer user can identify
parts automatically whenever special audio events happen e.g. where a person is speaking.
With Highlight Movie Creator user can combine all video events where a specific action
occurs and makes them into one video file. With the use of PATTERN add-on, the user can
discover and quantify behavioral patterns in the observational data. The recent add-on,
LogSqare records information about user activities and screen content in human-computer
interaction studies. More information can be found at http://www.mangold.de.
Comparison chart between qualitative data analysis tools which involve
video analysis
Observer
Interact
Transana
HyperResearch
Atlas.ti
Runs on Windows and MAC
W only
W only
√
√
W only
Integration of text with videos
√
√
√
√
√
Precise time codes for video events
√
√
√
√
Handling any multimedia file?
√
√
Mpeg
√
limited
Event recording
√
√
√
√
-
Method independent event
recording?
√
Annotate videos?
√
√
√
√
√
Multiple coding of videos?
√
√
√
Coding textual data
√
√
√
√
√
Organize coding systems
√
√
-
-
√
Export facility?
√
√
√
√
Add-on functions?
√
√
-
-
-
Statistical tools
√
√
-
-
-
Pattern analysis (add-on)
√
√
-
-
-
Video-tracking (add-on)
√
√
-
-
-
HCI logging (add-on)
√
√
-
-
-
Acquiring/integrating external data
(add-on)
√
√
-
-
-
14
Practice-Based Research
Bolt, B. (2006). A Non Standard Deviation: Handlability, Praxical Knowledge and Practice
Led Research, Speculation and Innovation: applying practice led research in the creative
industries., Queensland University of Technology. Accessed: 1/09/06. Available:
http://www.speculation2005.net
Frayling, C. et al (1997). (eds.) Practice- based Doctorates in the Creative and Performing Arts
and Design. N.p. [UK]: UK Council for Graduate Education.
Arts and Humanities Research Board (2000). Guide to the Research Grant Scheme.
http://www.ahrb.ac.uk/research/grant/guide.htm [accessed July 2000]
Harris. M. (1996). (ed.) Review of Postgraduate Education. N.p. [UK]: Higher Education
Funding Council for England. http://www.niss.ac.uk/education/hefce/pub96/m14_96.html
Hockey J. (2003). Practice–Based Research Degree Students in Art and Design: Identity and
Adaptation. Journal of Art & Design Education February 2003, vol. 22, no. 1, pp. 82-
91(10) The University of Gloucestershire, UK. Publisher: Blackwell Publishing on behalf of
the National Society for Education in Art and Design.
Biggs, M. (2003). The Role of "the Work" in Research. PARIP, University of Bristol, UK.
Accessed: 5/09/06. Available: http://www.bris.ac.uk/parip/biggs.htm
Gray, C. & Malins, J. (2004). Visualizing Research: A Guide to the Research Process in Art
and Design, Ashgate, Aldershot.
Reilly, L. 2002, An Alternative Model of 'Knowledge' for the Arts', Working Papers in Art
and Design, vol. 2. http://www.herts.ac.uk/artdes1/research/papers/wpades/vol2/reillyfull.html
Hannula, M. (2005) (ed) Artistic Research. Theories, Methods, and Practices, ISBN 951-53-
2743-1purchase at http://goart.gu.se/kf_publ/kf_publ.htm
Pakes, A., 2004, 'Art as Action or Art as Object? The Embodiment of Knowledge in Practice
as Research', Working Papers in Art and Design, vol. 3. Available:
http://www.herts.ac.uk/artdes1/research/papers/wpades/vol3/apfull.html
Scrivener, S. & Chapman, P., (2004).The Practical Implications of Applying a Theory of
Practice Based Research: A Case Study, Working Papers in Art and Design, vol. 3. Available:
http://www.herts.ac.uk/artdes1/research/papers/wpades/vol3/ssfull.html
Scrivener, S. (2002). The Art Object Does Not Embody a Form of Knowledge, Working
Papers in Art and Design,
http://www.herts.ac.uk/artdes1/research/papers/wpades/vol2/scrivenerfull.html
Sullivan, G. (2005). Art Practice as Research: Inquiry in the Visual Arts, Sage, California.
PBR Web links
http://www2.rgu.ac.uk/subj/ats/research/home.html
http://www.sunderland.ac.uk/~as0bgr/learnmat.html
http://www.point.ac.uk/
http://www.herts.ac.uk/artdes/research/papers/wpades/index.html
Practitioner Records
Edward Weston’s Daybooks: http://art-support.com/edwardweston.htm
Klee, Paul and Jurg Spiller. Paul Klee Notebooks: Volume 1: The Thinking Eye, Volume 2:
The Nature of Nature.London, 1992.
15
Reflective Practice
Schön, D. A. 1983, The Reflective Practitioner : How Professionals Think in Action, Basic
Books, New York.
Cowan, J. 1998, On Becoming an Innovative University Teacher: Reflection in Action.,
SRHE & Open University Press, London.
Holmes, A. 2006, ‘Reconciling Experimentum and Experientia: Ontology for Reflective
Practice Research in New Media’, Speculation and Innovation: applying practice led research
in the creative industries., Queensland University of Technology. Accessed: 1/09/06.
Available: http://www.speculation2005.net
Usability
Harker, S. (1995). The Development of Ergonomic Standards for Software, Applied
Ergonomics, 26,(4), pp275-279.
Thomas, C. and Bevan, N. (1995). Usability Context Analysis: A Practical Guide and
Performance Measurement Handbook EC Project Versions. National Physical Laboratory,
Teddington, Crown Copyright.
Other Useful References
Costello, B., Muller, L., Amitani, S. & Edmonds, E. (2005). Understanding the Experience of
Interactive Art: Iamascope in Beta_Space, Interactive Entertainment 2005, Creativity &
Cognition Studios Press, University of Technology Sydney, Australia, pp. 49-56.
Lawson, B. (2006) How Designers Think: The Design Process Demystified, 4th edn,
Architectural Press Elsevier, London.
Throop, J. C., (2003). Articulating Experience, Anthropological Theory, vol. 3, no. 2, pp. 219-
241.
Zeisel, J. (1981) Inquiry by Design : Tools for Environment-Behavior Research, Brooks/Cole
Pub. Co., Monterey, Calif.
Advice to PhD Students
http://www.uow.edu.au/research/rsc/hdrhb/PhDNotNobelPrize.pdf
It's a PhD, not a Nobel Prize - the most difficult thing is to get to grips with is how un-
amazing your PhD can be. This article presents the results of a study of experienced
examiners' attitudes.
http://www.phinished.org/ Peer support for finishing a PhD. Might strike some as a bit too
community/US-centric.
http://www.insidehighered.com/workplace/2005/11/30/tips - What They Don’t Teach You in
Graduate School
http://polaris.gseis.ucla.edu/pagre/network.html: Networking on the Network: A Guide to
Professional Skills for PhD Students - long article, but focuses on the things you should be
doing other than writing a thesis.
16
Appendix
Definitions and Terms
This part of the document is intended as a placeholder. The entries will be updated regularly.
Action Research
Action research grew out of attempts to acquire knowledge that would help change social
systems. It was essentially a theory-based approach grounded in real life that, in simplified
form, consisted of a cyclical process of conducting an investigation, taking action based on the
results of that enquiry, followed by evaluation of the improvements in the situation under
consideration. It has been developed further in organisational (Burke, 1994) and educational
research (Stringer, 2003) and different forms have evolved.
Action research requires intervention in order to study impact of change on a given situation
and thereby understand the situation under consideration. A number of forms have been
developed in different domains since its origins in the social sciences in the 1940s.
Action Research Electronic Reader:
http://www.scu.edu.au/schools/gcm/ar/arr/arow/default.html
Data Collection and Analysis Methods
Conversation analysis
The study of naturally occurring talk-in-interaction in order to discover how we produce an
orderly social world. CA provides an account of the machinery in operation within talk by a
fine-grained analysis of talk. It does not refer to context or motive unless they are explicitly
deployed in the talk itself. Conversation analysis has developed a highly sophisticated form of
transcription notation (q.v.) to support its fine-grained analysis.
Discourse analysis
A study of the way versions or the world, society, events and psyche are produced in the use
of language and discourse. The Foucauldian version is concerned with the construction of
subjects within various forms of knowledge/power. Semiotics, deconstruction and narrative
analysis are forms of discourse analysis. Further reading:
Hammersley, M. (2002), Discourse analysis: A Bibliographical Guide
http://www.cf.ac.uk/socsi/capacity/Activities/Themes/In-depth/guide.pdf
Drawing process analysis
A new kind of pen allows us to examine the drawing process in more detail. With a
commercial digital pen and special paper, we can record a designer's pen strokes, and use its
"instant replay" software to see how a picture was constructed.
Digital Sketching Website (in progress) http://www.uoregon.edu/~arch/digsketch/
Interviews
The interview technique is a systematic collection of verbal information. It consists in asking
about users opinions and attitudes to get basic information with prepared questions asked by
the interviewer. The answers are either written or recorded. The interviews can be structured
or unstructured. The terms structured interviews implies that the content of the interview, in
terms of the questions and their sequence, is predefined. Because of the structuring the
interview offers the opportunity for more systematic collection data. The unstructured
interview is more open-ended, and the interviewee develops the themes proposed by the
interviewer. Further reading:
http://www.ul.ie/~infopolis/methods/interv.html
Survey
Surveys are used to collect quantitative information about items in a population. Surveys of
human populations and institutions are common in political polling and government, health,
social science and marketing research. Survey may focus on opinions or factual information
17
depending on its purpose. On-line resource about how to conduct surveys:
http://www.managementhelp.org/commskls/surveys/surveys.htm
Observation
Observations can be conducted of individual behaviour or interactions among individuals, of
events or of physical conditions within a site or facility. They require well-trained observers
and detailed guidelines about whom or what to observe, when and for how long and by what
method of recording. The primary advantage of observation is its validity. When done well,
observation is considered a strong data collection method because it generates first-hand,
unbiased information by individuals who have been trained on what to look for and how to
record it. Observation does require time—for development of the observation tool, training of
the observers and the data collection—making it a more costly data collection method than
some of the others. Further reading:
http://www.acf.dhhs.gov/programs/ccf/resources/gbk_om/om_gbk_dcm.html
Protocol analysis
Protocol analysis is a rigorous methodology for eliciting verbal reports of thought sequences
as a valid source of data on thinking. It is used to gather data in usability testing in product
design and development, in psychology and a range of social sciences.Further reading:
Ericsson, K.A. & Simon, H.A. (1993), “Protocol Analysis: Verbal Reports as Data”,
Cambridge, MA: MIT Press.
Software for Data Analysis
The need for handling large qualities of observational data has given rise to software
applications that allow the researcher to collate the audio, video, text material and perform
various forms of qualitative and quantitative analysis. Examples include:
NVIVO: www.qsr.com.au
ATLAS.ti http://www.atlasti.de/
Observer (by Noldus: http://www.noldus.com/products/
HyperRESEARCH: http://www.researchware.com/
INTERACT: http://www.mangold.de/english/intlatest.htm
Video-cued Recall
Video-cued recall or retrospective reporting is a method for collecting verbal data commonly
used for investigating human cognitive processes. Because reports are made after the
experience, this method is regarded as having less impact on cognitive processes than
concurrent think-aloud methods. Reporting retrospectively, however, presents the risk that the
participant will forget details and that their recall will be interpretively filtered. The video
cued-recall method helps to avoid these pitfalls by using video to help the participant recall
the detail of their experience and avoid selective interpretation.
In order to support your claims, you need to provide an appropriate form of evidence.
Evidence can be acquired by three main approaches:
• Argumentation
• Proof
• Empirical
Evidence by Argument
• Argumentation
The act of forming reasons, making inductions, drawing conclusions, and applying them
to the case in discussion; the operation of inferring propositions, not known or admitted
as true, from facts or principles known, admitted, or proved to be true
Evidence by Proof
• Legal definition
Proof is the perfection of evidence, for without evidence there is no proof, although,
there may be evidence which does not amount to proof: for example, a man is found
murdered at a spot where another had been seen walking but a short time before, this
fact would be evidence to show that the latter was the murderer, but, standing alone,
would be very far from proof of it.
18
• Mathematical definition
Proof - a formal series of statements showing that if one thing is true something else
necessarily follows from it, as in Euclidean Geometry or Mathematical Logic
Empirical Evidence
Empirical evidence is needed for research that bases its findings on direct or indirect
observation as its test of reality. Evidence is acquired by:
• Observational Studies
• Situated studies
• People and technology interaction in context
• Practitioner accounts
• Personal reflections on activities and events
• Observer viewpoints
• Independent perspective on participant actions
Experimental Research
Experimental research involves testing hypotheses by manipulating variables within a
controlled situation
• Experimental Design
• Hypothesis Testing
• Controlled Variables
• Laboratory Environment
• Statistical Tests
• Empirical Evidence
Gödel’s theorem
http://en.wikipedia.org/wiki/Gödel's_incompleteness_theorem
Interactive Art and Research Practice
Studying art is recognised as a historical or critical scholarly activity rather than a natural
subject for field research. By its very nature, interactive art has particular characteristics that
necessitate a different form of inquiry to conventional areas of discourse in this field. The
involvement of the audience in the active experience of the work, for one thing, is a radical
departure from normal expectations of our relationship to art works. Some artists view
audience interaction as an integral part of the work itself and are not only keen to learn from
that behaviour but also wish to engage with the audience directly. In both audience and artist
collaborative experience, a process of evaluation takes place, an activity that requires
systematic forms of information, analysis and reflection. The evaluation of an emerging
interactive artwork or system is analogous with the development of an interactive software
system using user-centred design methods. In creative work there is a dual need: for best
results, the work should be carried out in as realistic (naturalistic) setting as possible and, at
the same time, the results should provide an opportunity to turn what is learnt into
modifications in the evolving art system. There are some important constraints that
differentiate the normal process of creating art from the research-oriented approach. Artists
working in a studio are in a natural setting for them but for research to be effective the
gathering of information is critical and this imposes constraints upon the way of working.
Artists working in a public space learn from the audience’s behaviour as they interact with the
new work. How they learn and discoveries that inform their work is both a new area of
creative practice and a source of knowledge for the wider community.
Practice-based Research
Practice-based Research is an original investigation undertaken in order to gain new
knowledge partly by means of practice. In a doctoral thesis, claims of originality and
contribution to knowledge may be demonstrated through creative outcomes in the form of
artefacts such as painting, music, designs, models, digital media, or creative events such as
performances, installations and exhibitions. Whilst the significance and context of that
knowledge is described in words, a full understanding of it can only be obtained with
reference to those outcomes. The textual description includes documentation of the research
process, as well as textual analysis or explanation to support its position and to demonstrate
critical reflection.
19
Practice-led Research
Practice-led research is concerned with the nature of practice and results in new knowledge
that has operational significance for that practice. Such research includes practice as an
integral part of its method and often falls within the general area of action research. The
results of practice-led research may be fully described in text without the inclusion of an
artefact. These are not practice-based doctorates of the type that include artefacts and works,
although the focus of the research can be to advance knowledge about practice, or to advance
knowledge within practice.
Practice-Based Research Methods
Practice-based researchers should devise a clear set of methods and techniques for collecting
data and analysing data. An important initial task is to identify the key elements of their
personal process which they intend to include in the data to be collected:
• Initial starting points or motivation for the project or work.
• Prior models or theories about how to create, perform or realise a creative artefact,
act or outcome…
• Time frame for the work or works to be created, performed, realised.
• Role of the creative artefact in the creative process.
• Environments and tools required to achieved the output
• Information to be gathered about the thinking, methods, tools, resources, support,
collaboration…
• Methods for collecting and collating data gathered
• Methods for analysing collated data
• Expected outcomes of the research process
• Relationship of the practice outcomes to the argument of the thesis.
Reflective Practice
The concept of reflective practice (Schön, 1983) provides a link between action research and
practice-based research. Schön is concerned with an individual’s reflection on his or her own
professional practice as distinct from the early forms of action research which were concerned
with situations more broadly. The combination of action research and reflective practice is an
approach widely adopted in educational research by teacher-researchers who might equally
call this form of research ‘practice-led’. Today a new generation of researchers in the creative
arts are pursuing both practice-based and practice-led research.
Usability Studies
ISO 9241 Usability Standard
“The extent to which a product can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use.” (Harker, 1995)
Provides a definition of the qualities of software systems in terms of the benefit to users, tasks
and organisational goals. Test criteria in the form of measures and metrics have been
developed from the ISO base criteria of task effectiveness and efficiency and user satisfaction
(Thomas and Bevan, 1995).
