Content uploaded by Paul Howard-Jones
Author content
All content in this area was uploaded by Paul Howard-Jones on Oct 28, 2014
Content may be subject to copyright.
This article was downloaded by: [University of Bristol]
On: 09 September 2014, At: 08:43
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Learning, Media and Technology
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/cjem20
Neuroeducational research in the
design and use of a learning technology
Paul Howard-Jonesa, Wayne Holmesa, Skevi Demetrioub, Carol
Jonesc, Eriko Tanimotod, Owen Morganc, David Perkinse & Neil
Daviese
a Graduate School of Education, University of Bristol, Bristol BS8
1JA, UK
b Cyprus Interaction Lab, Cyprus University of Technology,
Limassol, UK
c Chepstow School, Chepstow, UK
d Graduate School of Education, University of Bristol, Bristol, UK
e Duffryn High School, Newport, UK
Published online: 04 Sep 2014.
To cite this article: Paul Howard-Jones, Wayne Holmes, Skevi Demetriou, Carol Jones, Eriko
Tanimoto, Owen Morgan, David Perkins & Neil Davies (2014): Neuroeducational research
in the design and use of a learning technology, Learning, Media and Technology, DOI:
10.1080/17439884.2014.943237
To link to this article: http://dx.doi.org/10.1080/17439884.2014.943237
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or
howsoever caused arising directly or indirectly in connection with, in relation to or arising
out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
Neuroeducational research in the design and use of a learning
technology
Paul Howard-Jones
a
*, Wayne Holmes
a
, Skevi Demetriou
b
, Carol Jones
c
,
Eriko Tanimoto
d
, Owen Morgan
c
, David Perkins
e
and Neil Davies
e
a
Graduate School of Education, University of Bristol, Bristol BS8 1JA, UK;
b
Cyprus
Interaction Lab, Cyprus University of Technology, Limassol, UK;
c
Chepstow
School, Chepstow, UK;
d
Graduate School of Education, University of Bristol,
Bristol, UK;
e
Duffryn High School, Newport, UK
(Received 29 October 2013; accepted 7 July 2014)
Many have warned against a direct ‘brain scan to lesson plan’approach
when attempting to transfer insights from neuroscience to the classroom.
Similarly, in the effective design and implementation of learning technol-
ogy, a judicious interrelation of insights associated with diverse theoretical
perspectives (e.g., neuroscientific, pedagogical and classroom praxis) may
be required. A design-based research approach to the development of learn-
ing technology informed by neuroscience may be one way of achieving this
interrelation. Accordingly, here we report on some of the preliminary
research of a web app, known as ‘zondle Team Play’, that allows teachers
to teach whole classes using a games-based approach and which draws on
concepts from neuroscience. Rather than just exploring ‘what works’in
terms of the technology, low-fidelity prototyping and participant design
helped us explore aspects of praxis and affordances of the technological
design that were contingent upon each other. Five cycles of design, inter-
vention, analysis and reflection revealed some potential benefits of a neu-
roeducational approach to learning technology design, including the
development of related pedagogy, identification of immediate and future
neuroeducational research questions and the development of language
and terms suitable for communicating across interdisciplinary boundaries.
Keywords: teaching; pedagogy; games; technology; neuroscience; rewards
1. Introduction
The development of technology for learning is one field of innovation where the
new dialogue between neuroscience and education is considered closest to
having positive impact (Butterworth, Varma, and Laurillard 2011;Howard-
Jones et al. 2014; Royal Society 2011). However, it has been argued that the suc-
cessful integration of neuroscience into educational thinking and practice requires
© 2014 Taylor & Francis
*Corresponding author. Email: paul.howard-jones@bris.ac.uk
Learning, Media and Technology, 2014
http://dx.doi.org/10.1080/17439884.2014.943237
Downloaded by [University of Bristol] at 08:43 09 September 2014
aso-called‘neuroeducational’approach (Howard-Jones 2010) in which a transdis-
ciplinary collaboration between those working in education and neuroscience
assures optimal outcomes in terms of scientific validity and educational relevance.
The design of educational technology potentially introduces another field of
expertise and a new set of issues, requiring the integration of neuroscientific,
educational and technological concepts and understanding. Here, we argue
that this may best be achieved through a design-based research process invol-
ving low-fidelity prototyping and participant design, and we provide a case
study of our own [the design-based research of a games-based teaching app,
‘zondle Team Play’(zTP)] to illustrate the issues that can arise in working
across these three fields (education, neuroscience and technology).
2. Neuroeducational research and educational technology
The last decade has seen something of a step change in efforts to bring cognitive
neuroscience and education together in dialogue. This may be due to anxieties
over the ‘parallel world’of pseudo-neuroscience (Dekker et al. 2012; Geake
2008; Howard-Jones et al. 2009b), but it may also be because of new insights
arising from neuroscience with genuine value for education (Howard-Jones
2007; de Jong, Gog, and Jenks 2009; OECD 2007; Royal Society 2011).
Indeed, neuroscientists appear increasingly willing to speculate on the possible
relevance of their work to ‘real-world’learning, albeit from a vantage point on
its peripheries (e.g., Della Sala and Anderson 2012). Such speculation often
comes under the heading of ‘educational neuroscience’, a term that broadly
encompasses any cognitive neuroscience with potential application in edu-
cation. Accordingly, its research basis may be characterised by the epistem-
ology, methodology and aims of cognitive neuroscience. However, moving
from speculation to application is not straightforward, since the educational
value of insights from neuroscience rest on their integration with knowledge
from more established educational perspectives. Seeking meaningful relation-
ships between neural processes and the types of complex everyday learning
behaviours we can observe in classrooms presents a challenge.
One thing appears clear from the outset: a simple transmission model in which
neuroscientists advise educators on their practice, or developers on their pro-
ducts, is unlikely to be effective. Neuroscientists are rarely experienced in con-
sidering classroom practice, and neuroscience cannot provide instant solutions
for teachers. Instead, research is needed to bridge the gap between laboratory
and classroom. To emphasise the key role of educational values and thinking
in the design and execution of such a venture, researchers at the University of
Bristol have used the term ‘neuroeducational research’to describe this enterprise
(Howard-Jones 2010). For both scientists and educators, co-construction of con-
cepts requires broadening personal epistemological perspectives, understanding
different meanings for terms used in their everyday language (e.g., learning,
meaning, attention and reward) and appreciating each other’s sets of values
2P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
and professional aims. This boils down to having a dialogue about how the differ-
ent perspectives and their favoured types of evidence can inform about learning
in different but potentially complementary ways.
In contrast to such authentic interdisciplinary work, brief intellectual liaisons
between education and neuroscience are never likely to bear healthy fruit. These
flirtations may, indeed, spawn further neuromyths. A typical example of such
myth-making is when synaptic connections in the brain are used to explain
how we form connections between ideas. This often involves a conflation of
brain and mind that allows some educational practices to gain an apparently
neuroscientificflavour (research shows that explanations involving neuro-
science provide greater satisfaction, even when the neuroscience is irrelevant,
Weisberg et al. 2008). In reality, however, psychological theories about the
mind are key to understanding the significance of brain data for behaviours
such as learning; and association between ideas is a well-studied psychological
concept that is currently impossible to study at the level of the synapse.
Nevertheless, having this important conversation about how different per-
spectives inform learning is a first step towards a theoretical framework for
research at the interface of neuroscience and education. This can help us
combine findings more judiciously across perspectives to develop a better
understanding of learning. However, such an aspiration also has implications
for methodology. If there is a genuine commitment to interrelate findings
from component perspectives, the methods associated with these perspectives
should be adapted to better support such interrelation. For example, qualitative
interpretation of classroom discourse can draw usefully on neurocognitive con-
cepts in the interpretive analysis of its meaning. Some brain imaging studies can
contribute more meaningfully to the construction of neuroeducational concepts
if they include semi-structured interviews of participants, to derive experiential
insights about their constructs, strategies and attitudes. In some bridging
studies, judicious compromise and innovative approaches may help improve
the ecological validity of experimental tasks while still attempting to control
extraneous variables. Perhaps most unusually, researchers in the same team
may find themselves sequencing radically different methods to collect biologi-
cal, experiential and social evidence as they attempt to construct answers that,
collectively, help span the social–natural science divide.
We believe that using such answers in the design of educational technology
requires a similar process of integration. There is no guarantee that a useful
learning principle derived in the laboratory and demonstrated in the classroom
will be enhanced, or even survive, its implementation in a piece of software. To
ensure the best outcome, this implementation must occur through a design
process that includes potential end-users (i.e., teachers and learners) and
those who possess current understanding of the principle’s scientific basis
and the current limits of that basis. The need to include users, particularly
teachers, alongside other types of specialists in the design process, suggests a
participant design process as a natural extension of the neuroeducational
Learning, Media and Technology 3
Downloaded by [University of Bristol] at 08:43 09 September 2014
approach. To illustrate the advantages of this approach, we report here on the
design-based research of a web app, known as zTP, that enables teachers to
teach whole classes using a games-based approach.
zTP was developed iteratively with teachers in five cycles of design, inter-
vention, analysis and reflection. The design process involved a multidisciplin-
ary team and drew on neuroeducational theory, teachers’insights grounded in
practical classroom experiences and well-established design expertise. We
describe the process as design-based research, in which users are equal partners
of the design and development team. Our approach was closest to that described
by Facer and Williamson (2004)as‘informant design’, in which teachers are
seen as experts informing designers of key issues related to their experience,
helping to develop early design ideas and testing prototypes in development.
In this way, teachers had a critical role in shaping the design with their insights,
alongside those with neuroscientific and design expertise.
3. Towards a science of learning games
The use of digital games and games-based approaches to support learning,
especially on personal technologies such as smart phones and tablet computers,
has recently gained prominence (Koutromanos and Avraamidou 2014;
Richards, Stebbins, and Moellering 2013; Whitton 2014). The outcomes of
much games-based learning research have been affirmative (cf. Perrotta et al.
2013). However, whether it is possible to conclude that ‘studies have proven
empirically the efficacy of games-based learning over conventional methods’
(de Freitas and Maharg 2011, 20) remains arguable. In fact, games-based learn-
ing remains relatively uncommon in the classroom (Kenny and McDaniel 2011;
Wastiau, Kearney, and Van den Berghe 2009). Candidate explanations for this
lack of uptake include the paucity of robust evidence of the efficacy of games-
based approach to learning (Connolly et al. 2012), and the attitudes of teachers
towards the use of games in classrooms (Bourgonjon et al. 2013). Alternatively,
it may be due to a simple lack of individual access to appropriate technologies
in many schools (Games & Learning 2014). One approach to addressing this
last possibility may be to employ interactive whiteboards, which are widely
available in UK schools (Hennessy 2011) and which may afford a games-
based approach to whole-class teaching (in distinction to the more widely
researched games-based learning). Using interactive whiteboards may also
offer teachers more direct control of the games-based approach and may, there-
fore, prove more acceptable (Grady, Vest, and Todd 2013; Jackson 2009).
Another candidate explanation for the slow establishment of games-based
teaching and learning in schools may be the lack of a principled understanding
of related learning processes and pedagogy. In fact, one line of evidence suggests
the development of effective games-based teaching may arise from a carefully
considered interrelation of insights from diverse theoretical perspectives:
games-based learning, pedagogical, classroom praxis and neuroscientific
4P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
(Howard-Jones and Demetriou 2009). Fresh insight regarding the brain’s reward
system provides a rudimentary basis for understanding ‘engagement’provided
by games (‘engagement’is in fact a complex construct beyond the scope of
this paper, cf. Whitton 2011). Our motivation to win points in a game generates
signals in the brain’s reward system that are similar to those produced by our
attraction to many other pleasures such as food (Koepp et al. 1998). This activity
involves uptake of the neurotransmitter dopamine in the midbrain regions
(‘dopaminergic activity’). Primate studies show that a brief dopamine ‘spike’
will be generated simply by the awareness that a reward will certainly be pro-
vided (Figure 1(a)) or when a totally unexpected one is received (Figure 1(b)).
However, with the awareness that an uncertain reward may be provided (i.e.,
when uncertainty exists about whether a reward will be received or not), there
is a brief spike plus an additional ramping up of dopamine until the outcome is
known (Figure 1(c)) (Fiorillo 2003). Overall, this results in more dopamine
being released for uncertain rewards (represented by the area underneath the
lines in Figure 1), and this release peaks when the likelihood of receiving a
reward is 50%. This provides a potential neurobiological explanation for our
attraction to games involving chance (Shizgal and Arvanitogiannis 2003) and
suggested the approach developed in zTP.
While humans also appear most attracted to risks involving 50% uncertainty in
games, there is less attraction to this level of uncertainty when it is determined by
our own ability. One study shows that the level of certainty preferred by learners in
purely academic tasks is around 88% (Clifford and Chou 1991), a much higher
figure which is possibly due to the implications of failure for self- and social
esteem. However, working in the comfort zone of high certainty may not fully
involve the stronger motivational signals associated with the type of dopaminergic
activity observed in games (Koepp et al. 1998; Weinstein 2010). This may also
explain why emotional response during learning tasks has been found to increase
when these tasks are integrated into a chance-based game (Howard-Jones and
Demetriou 2009). Given thatemotional response can also support memory encod-
ing (LaBar and Cabeza 2006), we may also expect experiences involving more
emotional response to be more memorable.
Figure 1. Uptake of the neurotransmitter dopamine generated in response to the prob-
ability (P) of receiving a reward.
Learning, Media and Technology 5
Downloaded by [University of Bristol] at 08:43 09 September 2014
Combining learning with games of chance offers a potential way of increas-
ing reward signals and learning, without threatening esteem. There are many
examples in sport and in everyday life when success arises from a combination
of ability and chance, and well-matched competition (i.e., with around 50%
likelihood of outcome, such as a football game) provides a highly engaging
challenge. Children, especially boys, appear to prefer the inclusion of
chance-based uncertainty in learning tasks (Howard-Jones and Demetriou
2009). Importantly for education, a positive relationship between reward
activity in the brain and memory formation has also been demonstrated. In
an educational game, dopaminergic activity due to gameplay rewards was esti-
mated (based on the extent of expected gain) for each round. This signal pre-
dicted the success of memory recall more effectively than the size of the
reward itself (Howard-Jones et al. 2009a).
A previous classroom study has also shown that mediating rewards for
learning with chance-based events can affect the discourse around learning in
positive ways (Howard-Jones and Demetriou 2009). It tends to encourage
open motivational talk and allows students to introduce a self-serving bias
that attributes failure to chance (thus minimising challenges to self-esteem)
and success to ability. Most recently, an independent group of researchers
have extended investigations of reward uncertainty and demonstrated a clear
link between increased motivation and improved learning in response to uncer-
tain rewards (Ozcelik, Cagiltay, and Ozcelik 2013).
The following key points arise from the neuroscientific and neuroeduca-
tional research for educational practice with learning games and comprised
the starting point for the development of zTP
.Learning games can increase student engagement through inclusion of
chance-based components that increase the uncertainty of rewards for
learning (Ozcelik, Cagiltay, and Ozcelik 2013).
.The brain’s response to rewards can be very brief (Bogacz et al. 2007).
That suggests a close intermingling of learning and gameplay elements
is needed for the gameplay to support the learning.
.Anticipation of an uncertain reward is likely to generate a more extended
‘window of enhanced attention’or ‘teachable moment’(Howard-Jones
and Demetriou 2009).
.Avoiding a loss does not generate the same reward signals as a gain,
suggesting a generally positive scoring system may best support motiv-
ation (Howard-Jones et al. 2010).
4. Design-based research process
Our design team comprised two academic researchers (who between them had
experience in neuroscience research, psychology, education and games-based
learning), two postgraduate assistants, teachers from two comprehensive
6P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
schools in South Wales (hereinafter referred to as School A and School B) and
an experienced software developer (Doug Lapsley at zondle).
The team used a design-based research approach, simultaneously pursuing
practical innovation and theory building by means of the iterative development
of solutions in a real-world situation (Brown 1992; Cobb et al. 2003). The study
did not seek to verify the hypothesised role of neural processes. Instead, it set
out to build upon an understanding of those processes, as derived from studies
of more controlled environments, by means of the iterative design of an inter-
vention (a teachers’interactive whiteboard app) and its use within the real-
world context of a conventional classroom. In short, the study was exploratory
and developmental rather than evaluative. The science of learning games out-
lined above was the starting point for the first iteration of what eventually
became the app known as zTP. After this, reflection on observations and out-
comes in the classroom were the driving forces for developing both the
further design of the app and good practice. Rather than a prescription for class-
room practice, concepts about the brain provided a useful starting point for
innovation and contributed to a helpful framework for stimulating reflection
and understanding.
The process comprised five cycles of design, intervention, analysis and
reflection (the final cycle is ongoing) and, for purposes of analytical data tri-
angulation, involved various methods of data collection (observations, video
recordings, interviews and group discussions, the balance between these
methods evolving from cycle to cycle). In each research cycle, rather than eval-
uating the intervention (by comparing its effectiveness with non-gameplay
approaches), we set out to identify instances of apparent learning gain as critical
examples that could inform discussions and reflection about pedagogy and sub-
sequent cycles. In the first and fourth interventions, learning gains were ident-
ified by means of a written pre- and post-test. In the second and third
interventions, when we were focusing on a group with low-literacy ability,
we used a non-written measure. In the first four interventions, two digital
video cameras (facing class and teacher) recorded the session, the video record-
ings being used as a basis for subsequent discussion and group analysis.
Informed consent was given by the parents of all participating students.
Accumulated insights helped generate a fully operational app for the fifth iter-
ation, which is currently the focus of further laboratory-based studies involving
neuroimaging.
4.1. Design cycle 1
4.1.1. Design
Based on the science of learning games outlined above, the team developed a
low-fidelity prototype game using Microsoft PowerPoint. The presentation
comprised a repeating pattern of one to two slides of content, on the topic of
Learning, Media and Technology 7
Downloaded by [University of Bristol] at 08:43 09 September 2014
‘reproduction’, followed by one to two slides of multiple-choice questions that
assessed knowledge of this content in which each answer was labelled with one
of four colours. A ‘student response system’was also developed, comprising
sets of four 15 cm square coloured cards (the same colours as those used to
label the potential answers), hinged with tape, one set for each student. The
final part of the design was a television quiz-style ‘wheel of fortune’, divided
into coloured sectors.
4.1.2. Intervention
The first intervention took place in School A, with 25 students in a Year 7
science class (mean age 11 years 6 months; 13 males, 12 females), and the
assessed learning objectives focused on the acquisition and recall of knowledge
rather than understanding. For around 10 minutes, the teacher taught an aspect
of the topic ‘reproduction’using the PowerPoint slides to structure and illus-
trate. The teacher then revealed one of the multiple-choice questions. To
respond, the students had to choose an answer and note its colour on the Power-
Point slide, and fold their squares so that that colour faced frontwards.
This approach to using multiple-choice questions in whole-class teaching
was so far conventional. However, chance-based uncertainty, as suggested by
the neuroscience (Fiorillo 2003), was introduced to mediate the receipt of
rewards. Each correct answer was rewarded with the option to receive a
point, represented by a counter, or to take a chance and receive either zero or
two points based on a spin of the ‘wheel of fortune’. This became known as
‘gaming the points’.
4.1.3. Analysis and reflection
Throughout the session the students, particularly the boys, were observed to be
engaged by this novel approach to teaching and learning: they were observed to
be animated and clearly excited by the challenges, absorbed in the activity and
enjoying the immediate feedback, and attending closely to the teacher’s talk
(Whitton 2011). However, this first session also highlighted how engagement
does not necessarily translate into learning (Whitton 2014). Mean scores out
of a possible 14 marks (with standard deviations in parentheses) for pre- and
post-tests were 4.6 (2.4) and 5.8 (3.0), which represents only a modest improve-
ment (a Wilcoxon non-parametric signed ranks test showed that this outcome
was statistically significant: z=−2.82, p= .005, r=−0.40). Nevertheless, the
data helped identify instances of apparent learning gain and the outcome
encouraged the researchers to proceed to the next cycle of design.
All video recordings, in this and subsequent research cycles, were coded
informally and iteratively by the academic team, focusing on teacher talk
(e.g., asking a factual question, checking understanding, motivating, praise,
feedback and classroom management), student talk (centred on the question,
8P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
centred on the gameplay, directed at the teacher, with other students and
suggesting engagement), teacher actions (delivery of the content and game
elements) and student actions (attending to the teacher, animation/excitement
and attending to the game). The recordings were also used to stimulate recall
in discussions with the class teachers.
The video recording from this session revealed that notable moments of
heightened attention occurred when the correct answer was about to be
announced and the wheel of fortune was turning: in other words, as the students
were about to find out whether they would gain some points. However, this
meant that the attention of the students was mostly on the game rather than
the learning content. In addition, the teacher tended to indicate the correct
answer quickly and first, such that the putative ramping up of dopaminergic
activity was not being fully exploited for learning. Accordingly, it was clear
that, if any engagement fostered by the gameplay was to be of educational
value, greater effort would have to be made to ensure the learning content
was more closely associated with the gameplay.
Nevertheless, although least successful as an intervention, a great deal was
learned from this session. For example, a post-interview with the teacher con-
firmed that, because they needed to divide their attention between game hosting
and teaching to teaching, this novel approach to teaching adds to their cognitive
load rather than reducing it. Also, while the conventional scaffolding strategies
remain crucial to learning (e.g., checking understanding through verbal
exchanges or providing hints that focus minds on relevant content), it was
clear that they can easily be forgotten in the heat of the game. If not harnessed
correctly, the excitement of the game can distract students and teacher from the
learning rather than help them engage with it.
In summary, this first design cycle suggested that if the approach were to be
successful the teacher needed to implement three principles. They should
(1) give the students time to consider their responses before revealing the
correct answer;
(2) give support to help further the students’understanding of the learning
content when they were answering the questions;
(3) discuss potential misconceptions as the answers were being revealed
(e.g., ‘If this was your answer, you may have forgotten that ….’), so
that those who answered incorrectly could receive additional instruction
during this brief window of apparent heightened engagement.
4.2. Design cycle 2
4.2.1. Design
In the second iteration of the design, a new topic was chosen. The content of the
PowerPoint slides focused on the understanding, rather than the recall, of the
Learning, Media and Technology 9
Downloaded by [University of Bristol] at 08:43 09 September 2014
grammatical concepts of noun, pronoun, verb and tense. Otherwise, the design
was unchanged (the student response system and wheel of fortune, for example,
were as before).
4.2.2. Intervention
This second intervention took place in School B, involving an experienced
teacher of literacy and a Year 9 group of 12 students (mean age 13 years and
7 months; 8 males and 4 females) receiving additional support for literacy.
At the beginning of the session, this low-literacy group undertook a pretest
of five questions, focusing on their understanding and application (rather than
their knowledge) of grammatical concepts such as noun, pronoun, verb and
tense. The outcomes were later used to identify instances of apparent learning
gain, with responses within the games-based session being compared with
responses in the pretest.
In most respects, delivery of the game followed the pattern of the first inter-
vention. However, based on the previous outcomes, in this cycle there was a
greater emphasis on the educational content rather than the gameplay (with
three to four slides of content followed by the one to two slides of questions).
In addition, the teacher aimed to implement the principles derived from the pre-
vious design cycle.
This second intervention introduced a further development based on neuro-
science research, this time around the relationship between the brain’s reward
response and social context. This research suggests (i) a link between midbrain
dopamine uptake and the expectations generated by recent history (Schultz
1998) and (ii) that the maximum uptake is proportional to the maximum
reward available in a context (Nieuwenhuis et al. 2005). Drawing on this
research, in this second intervention, the teacher was encouraged to increase
gradually the number of points available for each round.
4.2.3. Analysis and reflection
The mean of pretest learning scores was 53%, and mean scores during the game
(which used questions of similar type to the pretest) was 65%. Although statisti-
cal analysis was inappropriate (the sample size was small and the answering of
questions was occasionally supported by the teacher), the data again helped
identify instances of apparent learning gain, with a clear example of four stu-
dents who failed some pretest questions but who answered correctly similar
questions during the game. The discourse that appeared to prompt this high-
lighted how the additional engagement that the game was intended to
provide may be used as an opportunity to scaffold students’learning.
The video recording revealed that there were also several instances of the
teacher checking understanding and praising it, and of students being supported
as they were answering questions. Generally, however, the teacher was
10 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
disappointed at not being able to apply consistently the three teaching principles
arising from the first iteration. Although an experienced teacher, she found that
the game format of the lesson made additional demands on her management and
thinking processes and took up time (e.g., giving out counters and moving back
and forth between whiteboard and wheel of fortune).
Nevertheless, the students were observed to be highly animated and
engrossed throughout the session, with the continual raising of the stakes
appearing to help maintain the students’excitement and motivation to partici-
pate (while possibly also reducing the likelihood of engagement being dimin-
ished by expectation). The increasing stakes also made the final outcome
even less predictable, since later rounds had more influence on scores than
earlier ones.
The chance-based outcomes also appeared to generate emotional teacher–
student empathy whatever the outcome, suggesting that games-based teaching
may change the emotional content of teacher–student exchanges. When out-
comes arose through chance, the teacher could acknowledge failure as expres-
sively and as strongly as success. This may make for a more authentic sharing of
emotions than afforded by the conventional classroom focus on the positive.
There was also, for example, some rejoicing when someone else in the class
failed to win points. A recent fMRI study, carried out in order to provide
insight into such issues, revealed that the reward response to our competitor
is related to their losses (Howard-Jones et al. 2010).
4.3. Design cycle 3
4.3.1. Design
The third iteration focused on reducing the demands on the teacher by automat-
ing part of the game (that is on classroom pragmatics rather than neuroscientific
theory). A purpose-built macro for PowerPoint was developed (Figure 2).
While students still gave their responses using coloured cards, the macro
allowed the teacher to record on-screen the students’responses to questions
and their decisions whether or not to game their score. It also included an auto-
matic flashing light version of the wheel of chance that automatically recalcu-
lated the scores. Although technology-based, we would still describe this
prototype as low-fidelity, with minimal functionality and graphic quality.
4.3.2. Intervention
The third intervention involved the same participants as the second interven-
tion, studying another set of grammatical concepts: adjectives, adverbs, capitals
and full stops in sentences, commas and speech marks. This time, however, par-
ticipating students were divided into pairs and competed as six teams, in order
to give an opportunity for additional collaborative and constructivist dialogue to
Learning, Media and Technology 11
Downloaded by [University of Bristol] at 08:43 09 September 2014
support learning. The teacher again aimed to implement the teaching principles
identified earlier and also was encouraged to adopt a new strategy to take further
advantage of the supposed ramp in dopaminergic activity (based on Fiorillo
2003): revealing the incorrect answers, and explaining why they were incorrect,
before revealing the correct answer: that is to say, while the students were most
attentive as they waited to find out whether or not they had been successful.
4.3.3. Analysis and reflection
The mean of pretest learning scores across teams was 39%, while the mean
scores during the game were much higher, at 80%, broadly suggesting that
some learning had taken place (although, for the same reasons as before, stat-
istical analysis was again inappropriate). An important outcome of introducing
a technology that removed the need to manage counters was a four-fold increase
in student–teacher interactions, of which a higher proportion were related to the
learning content. Initially, students responded in brief to teacher questions.
After around 10 minutes, exchanges grew in duration and complexity, concepts
and principles were discussed, and students offered unprompted examples and
asked questions to verify understanding.
In the post-interview, the teacher indicated that she was much more positive
than in the previous cycle. She felt she had been able to focus more on the teach-
ing and was delighted with the level of engagement that she believed the game
had helped create. In addition, she noted that students had shown signs of inde-
pendent thinking about the principles, and that the discussion became more
spontaneous as students made unprompted contributions. Meanwhile, the strat-
egy of revealing incorrect answers before correct answers appeared to increase
Figure 2. Screenshot of PowerPoint with additional interactivity for gameplay pro-
vided by a small macro programme.
12 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
further the students’attention. The video recording revealed that the intense
engagement was particularly evident for several students who became increas-
ingly animated as the teacher scrutinised each incorrect option in turn, taking
full advantage of these key ‘teachable moments’(as suggested by the notion
of dopaminergic activity) while building up the tension until the correct
answer was finally revealed.
4.4. Design cycle 4
4.4.1. Design
The fourth iteration used unaltered the PowerPoint macro (including the auto-
matic flashing light version of the wheel of chance) and coloured cards devel-
oped in the previous cycles. A new topic area was chosen (the evaluation of
plastic products).
4.4.2. Intervention
The sample was a mixed-ability Year 10 Design and Technology group (in School
A) comprising 9 students (mean age 15 years 7 months; all males). The teacher
believed, based on their knowledge of the students and their own professional
experience, that the students would benefit from independent and constructivist
learning opportunities. Accordingly, the intervention approach was slightly
revised. General concepts were presented at the beginning of the lesson, without
the on-screen interface, and discussed with the students. Then, working in teams
of two (comprising four pairs of students and one student supported by a classroom
assistant), the students used notes provided by the teacher to support discussion,
knowledge construction and joint decision-making. After 12 consecutive game
rounds, involving the flashing light ‘wheel of chance’, notes were removed and
the students faced another 12 rounds. Incorrect responses to the questions were
used to identify potential issues with understanding and these prompted additional
explanations from the teacher and teacher–student discourse to scaffold learning.
This change in approach was mostly a response to the needs ofthe particular learn-
ing context; and its implementation drew attention to howa games-based approach
to teaching, like other types of teaching, is situated in contextual issues such as
group dynamics, ability, level and topic.
A pre- and post-test required the students to choose an appropriate type of
plastic with which to manufacture a specified product, by recalling and correctly
applying principles discussed in the session.
4.4.3. Analysis and reflection
Mean scores out of a possible five marks (with standard deviations in parenth-
eses) for the pre- and post-tests were 1.3 (0.8) and 3.2 (1.3) (despite the small
Learning, Media and Technology 13
Downloaded by [University of Bristol] at 08:43 09 September 2014
sample size, a Wilcoxon non-parametric signed ranks test showed that this
outcome was statistically significant: z=−2.49, p= .013, r=−0.89). The differ-
ence in the mean scores represents a large effect size (Cohen’sd= 2.18) which,
and despite the inevitable confounding variables, suggests that some learning
was taking place. This was reaffirmed by the teacher, who in the post-interview
argued that the students had achieved ‘good’levels of understanding.
Student talk during the session included a small number of queries to the
teacher, publicly expressed gameplay talk (boasting, teasing and joking) and
many furtive utterances as they quietly conferred with their partner. The confer-
ring (when it was audible in the video recording) was chiefly about learning
content and gameplay strategy. Often, during these exchanges, students main-
tained their visual attention on the teacher and question displayed on the screen
at the front of the class, as if trying to conceal their conversation from the rest of
the class. When announcing answers, the teacher revealed incorrect answers
first in order to exploit the window of attention or ‘teachable moment’
created by anticipation. Both quiet conferring and public exclamations indi-
cated close attendance to this information. There were several occasions
when the teacher’s talk slipped into something resembling that of a television
game-show host. This appeared to generate more excitement, working up the
emotions of the players, sometimes gentle goading, sometimes a voice of
caution or comfort.
Those who were not in the lead towards the end of the lesson took all oppor-
tunities to game their scores, as their chances to win without doing so dwindled.
Other strategies included teams avoiding giving away answers by hiding their
response until the last minute, by not putting it up or covering it with their bag,
etc. Some went as far as beginning with an answer they knew was wrong and
encouraging others to see it, before changing it at the last moment to their
chosen response. This prompted the researchers to consider the potential
benefits of an electronic response system instead of the coloured cards. This
may allow responses to be covert until all students had committed themselves,
so preventing plagiarism. It would also reduce the time taken for responses to be
collected and liberate the teacher from some of the remaining administrative
tasks that teaching with the game involved, allowing the teacher to focus
their attention more entirely on the teaching.
4.5. Design cycle 5
The fifth iteration of this games-based approach to teaching involved the colla-
borative design of a web app, in association with the developers of a games-
based learning platform, ‘zondle’. The app, known as ‘zTP’(Figure 3), was
itself designed iteratively: it was based on the low-fidelity versions discussed
above and scaffolded by a series of conversations between this paper’s lead
authors and the developer. No additional neuroscientific insights were incorpor-
ated. Instead, the aim was to make the app robust, easier to use and more widely
14 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
available than the low-fidelity prototypes. The alpha version of the web app was
further mediated by extensive feedback from users both in the UK and from
overseas (including the USA, Croatia and Australia). The app is freely available
on the developers’website (www.zondle.com) and is discussed in some detail
in Howard-Jones and Fenton (2012).
zTP was designed to be used on any interactive whiteboard (or with a com-
puter and projector) that has internet access. Teachers can import their Power-
Point slides into the system and can write appropriate multiple-choice
questions. Alternatively, they can use and, if they choose, amend any of the
more than 12,000 zTP topics written by other teachers and currently on the
system. The app provides a way to allocate answers to teams, automatically
allocates and records points, and includes a wheel of chance that can be
started by a student swiping the interactive whiteboard. Finally, students can
interact directly with the app by using any mobile device or computer with
internet access (much as if using an electronic response system), enabling stu-
dents in different locations to compete in a single zTP session (e.g., a class in
Croatia and a class in the USA have competed in several zTP sessions,
which has led to further collaboration between the schools involved).
Examples of suggestions made by teachers who used the alpha version of the
app, which were incorporated into the current beta version of the app, include
showing a thumbnail of the current learning content slide (to help orientate the
teacher), the ability to switch the interface left to right (so that teachers can
easily interact with it whichever side of the display that they prefer to stand)
Figure 3. The main screen from the app ‘zTP’, showing a question about the Tudors.
Learning, Media and Technology 15
Downloaded by [University of Bristol] at 08:43 09 September 2014
and the ability to hide answers given by the teams until all the teams have
answered (to minimise teams copying each other). Anecdotal evidence for
the playability, practicality and effectiveness has so far been positive: ‘it was
insightful watching the children in different groups –listening to their
thought processes and how they decided on their answers’(Hallybone 2012).
However, zTP is currently the focus of further laboratory-based studies, includ-
ing the use of neuroimaging, which will be reported later.
5. Discussion
5.1. Pedagogy and classroom praxis, not just product
The development of zTP has shown that there are potential benefits from a
games-based teaching app grounded in neuroscientific research. However, it
has also highlighted the importance of simultaneously developing an under-
standing of how such technology is best implemented, including the associated
pedagogy. Such implementation and pedagogy were informed by the partici-
pants (the teacher and student experiences) but also by considering the scientific
principles involved. This, perhaps, is true of other types of educational technol-
ogy but in the case of novel approaches informed by neuroscience there may be
a special case for ensuring transdisciplinary construction of associated pedago-
gic principles, given the distance between biological and educational perspec-
tives on learning.
Construction of pedagogical understanding was not simply important for
the implementation of the app, but also fed back into its design. The
implementation of our low-fidelity prototype allowed us to understand the
potential importance of the teacher scaffolding learning just prior to students
gaming their points through a combination of educational insight (feedback
from teacher and students in the classroom) and neuroscientific understanding
(in terms of the ‘ramping up’of dopamine shown in Figure 1(c), Fiorillo
2003). This insight influenced how incorrect answers can be revealed in the
most recent zTP version. As students were expected to be highly engaged
(attentive) during this period, incorrect options disappear as the teacher
touches them, allowing the teacher to provide a structured dismissal of
these options as they talk through why each one is incorrect. Another
example was the need for the teacher to be able to raise and lower stakes
spontaneously through the game. Again, this arose through a combination
of educational insight and neuroscientific understanding (in terms of effects
of the points available on midbrain dopamine response, Shizgal and Arvani-
togiannis 2003). Thus, a convenient way for the teacher spontaneously to
raise the stakes in each round was introduced into the design. Our process
allowed pedagogy and product design to come about together and inform
each other’s development, supporting the potential for their optimal inter-
relationship in the classroom.
16 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
5.2. Demand push, not just technology pull
Working with low-fidelity prototypes prevented trends in technology, the ‘lure
of the new’, from dictating the design of the outcome. Instead, having been
based on current neuroscientific understanding, this technology was then
shaped by the needs and wishes of teachers and students (rather than by, for
example, the capabilities of conventional audience response systems or the
immersive approach of much games-based learning).
5.3. Translating neuroscientific principles to the classroom
There were a large number of usability and pedagogical issues encountered
during the design of zTP (e.g., how to show the next content slide to the
teacher without showing it to the students, how to prevent students observing
and copying each other’s responses and how to allow the teacher to work
through the incorrect answers before revealing the correct answer to take advan-
tage of the ‘teachable moment’). These issues were all addressed by referring
back to the original neuroscience research to inform the learning principles,
the feedback of the teachers and students to confirm the pedagogy and class-
room pragmatics, and the experience of the research and design team to deter-
mine the final implementation. This approach appears to have been successful,
and it highlights why attempts to generate technology on the basis of sound neu-
roscientific learning principles are unlikely to come to fruition in the classroom
without a participant approach to design.
The design-based research process described above allowed us to combine
the neuroscience and educational insights, to identify and develop both an
effective piece of technology and the pedagogy required to implement it. Our
design process did not set out to evaluate the general educational value of prin-
ciples which had been studied in the laboratory and through quasi-experimental
classroom studies. Nor are we able to make claims about the efficacy of this
teaching game compared with other types of teaching strategy. We do,
however, claim that it emphasises the need for technology based on neuroedu-
cational concepts to be developed using a similarly interdisciplinary approach
as should be used to develop the concepts themselves.
Finally, we are pleased to report that, to date, more than 35,000 zTP sessions
have been played by users from more than 30 countries worldwide. However,
we should emphasise that the present and all future versions of zTP will always
be limited by our current state of scientific knowledge, which grows daily but
will always be partial. Certainly, at the time of writing, many fundamental
scientific and educational questions still require further research, such as the
exact mechanisms by which midbrain dopamine accelerates learning, how
the games-based approach may work if used over extended periods and how
suitable it may be for different contexts (such as those involving different abil-
ities, age groups, topics and gender). We look forward to tackling and reporting
on these and other issues in the future.
Learning, Media and Technology 17
Downloaded by [University of Bristol] at 08:43 09 September 2014
Notes on contributors
Paul Howard-Jones is Reader in Neuroscience and Education at the Graduate School of
Education in the University of Bristol (UK).
Wayne Holmes is a Researcher at the London Knowledge Lab (Institute of Education,
University of London, UK) and teaches education and technology at the Graduate
School of Education (University of Bristol). He is also Head of Education for zondle.
Skevi Demetriou is a researcher and lecturer in the Department of Communications and
Internet Studies at the Cyprus University of Technology (Limassol, Cyprus).
Carol Jones is the former Special Educational Needs Co-ordinator at Chepstow School,
Chepstow (UK).
Eriko Tanimoto is a practising teacher of Design and Technology in the South of
England (UK), with a masters in Psychology of Education Owen Morgan is an assistant
headteacher leading learning and progress and the current Special Educational Needs
Co-ordinator at Chepstow School, Chepstow (UK).
David Perkins is Head of the History Department at Duffryn High School, Newport
(UK).
Neil Davies is Assistant Headteacher at Duffryn High School, Newport (UK).
References
Bogacz, R., S. M. McClure, J. Li, J. D. Cohen, and P. R. Montague. 2007. “Short-Term
Memory Traces for Action Bias in Human Reinforcement Learning.”Brain
Research 1153: 111–121.
Bourgonjon, J., F. D. Grove, C. D. Smet, J. Van Looy, R. Soetaert, and M. Valcke.
2013. “Acceptance of Game-Based Learning by Secondary School Teachers.”
Computers & Education 67: 21–35.
Brown, A. L. 1992. “Design Experiments: Theoretical and Methodological Challenges
in Creating Complex Interventions in Classroom Settings.”Journal of the Learning
Sciences 2 (2): 141–178.
Butterworth, B., S. Varma, and D. Laurillard. 2011. “Dyscalculia: From Brain to
Education.”Science 332 (6033): 1049–1053.
Clifford, M. M., and F.-C. Chou. 1991. “Effects of Payoff and Task Context on
Academic Risk Taking.”Journal of Educational Psychology 83 (4): 499–507.
Cobb, P., J. Confrey, A. diSessa, R. Lehrer, and L. Schauble. 2003. “Design
Experiments in Educational Research.”Educational Researcher 32 (1): 9–13.
Connolly, T. M., E. A. Boyle, E. MacArthur, T. Hainey, and J. Boyle. 2012. “A
Systematic Literature Review of Empirical Evidence on Computer Games and
Serious Games.”Computers & Education 59: 661–686. doi:10.1016/j.compedu.
2012.03.004.
De Freitas, S., and P. Maharg. 2011. “Digital Games and Learning: Modelling Learning
Experiences in the Digital Age.”In Digital Games and Learning, edited by S. de
Freitas and P. Maharg, 17–41. London: Continuum.
18 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
Dekker, S., N. C. Lee, P. Howard-Jones, and J. Jolles. 2012. “Neuromyths in
Education: Prevalence and Predictors of Misconceptions among Teachers.”
Frontiers in Psychology 3. doi:10.3389/fpsyg.2012.00429.
Della Sala, S., and M. Anderson. 2012. Neuroscience in Education: The Good, the Bad
and the Ugly. Oxford: Oxford University Press.
Facer, K., and B. Williamson. 2004. Designing Educational Technologies with Users.
A Handbook from Futurelab. Bristol: Futurelab.
Fiorillo, C. D. 2003. “Discrete Coding of Reward Probability and Uncertainty by
Dopamine Neurons.”Science 299 (5614): 1898–1902.
Games & Learning. 2014. Teachers Surveyed on Using Games in Class. Initial Report.
Accessed June 12. http://www.gamesandlearning.org/2014/06/09/teachers-on-
using-games-in-class/
Geake, J. 2008. “Neuromythologies in Education.”Educational Research 50 (2):
123–133.
Grady, S. E., K. M. Vest, and T. J. Todd. 2013. “Student Attitudes Toward the Use of
Games to Promote Learning in the Large Classroom Setting.”Currents in
Pharmacy Teaching and Learning 5 (4): 263–268.
Hallybone, D. 2012. From Dawn till Dusk –Games and Life. Musings on ICT,
Handhelds, Learning and Life. Zondle Team Play. Accessed June 15, 2014.
http://hallyd.edublogs.org/2012/07/17/zondle-team-play/
Hennessy, S. 2011. “The Role of Digital Artefacts on the Interactive Whiteboard in
Supporting Classroom Dialogue.”Journal of Computer Assisted Learning 27 (6):
463–489.
Howard-Jones, P. 2007. Neuroscience and Education: Issues and Opportunities.
London: Teaching and Learning Research Programme, ESRC.
Howard-Jones, P. 2010. Introducing Neuroeducational Research: Neuroscience,
Education and the Brain from Contexts to Practice. Abingdon: Routledge.
Howard-Jones, P., R. Bogacz, S. Demetriou, U. Leonards, and J. Yoo. 2009a. “From
Gaming to Learning: A Reward-Based Model of Decision-Making Predicts
Declarative Memory Performance in a Learning Game.”British Psychological
Society Annual Conference, Brighton.
Howard-Jones, P. A., R. Bogacz, J. H. Yoo, U. Leonards, and S. Demetriou. 2010.
“The Neural Mechanisms of Learning from Competitors.”NeuroImage 53 (2):
790–799.
Howard-Jones, P. A., and S. Demetriou. 2009. “Uncertainty and Engagement with
Learning Games.”Instructional Science 37 (6): 519–536.
Howard-Jones, P., and K. Fenton. 2012. The Handbook of Teaching with Gaming.
Raleigh, NC: Lulu Press.
Howard-Jones, P., L. Franey, R. Mashmoushi, and Y.-C. Liao. 2009b. “The
Neuroscience Literacy of Trainee Teachers.”British Educational Research
Association Annual Conference, University of Manchester, Manchester.
Howard-Jones, P., M. Ott, T. van Leeuwen, and B. De Smedt. 2014. “The Potential
Relevance of Cognitive Neuroscience for the Development and Use of
Technology-Enhanced Learning.”Learning, Media and Technology 1–21.
doi:10.1080/17439884.2014.919321.
Jackson, J. 2009. “Game-based Teaching: What Educators Can Learn from
Videogames.”Teaching Education 20 (3): 291–304.
de Jong, T., T. Gog, and K. Jenks. 2009. Explorations in Learning and the Brain: On
the Potential of Cognitive Neuroscience for Educational Science. New York:
Springer.
Learning, Media and Technology 19
Downloaded by [University of Bristol] at 08:43 09 September 2014
Kenny, R. F., and R. McDaniel. 2011. “The Role Teachers’Expectations and Value
Assessments of Video Games Play in Their Adopting and Integrating Them into
Their Classrooms.”British Journal of Educational Technology 42 (2): 197–213.
Koepp, M. J., R. N. Gunn, A. D. Lawrence, V. J. Cunningham, A. Dagher, T. Jones,
D. J. Brooks, C. J. Bench, and P. M. Grasby. 1998. “Evidence for Striatal
Dopamine Release During a Video Game.”Nature 393 (6682): 266–268.
Koutromanos, G., and L. Avraamidou. 2014. “The Use of Mobile Games in Formal and
Informal Learning Environments: A Review of the Literature.”Educational Media
International 51 (1): 49–65.
LaBar, K. S., and R. Cabeza. 2006. “Cognitive Neuroscience of Emotional Memory.”
Nature Reviews Neuroscience 7 (1): 54–64.
Nieuwenhuis, S., D. J. Heslenfeld, N. J. Alting von Geusau, R. B. Mars, C. B. Holroyd,
and N. Yeung. 2005. “Activity in Human Reward-Sensitive Brain Areas Is Strongly
Context Dependent.”NeuroImage 25 (4): 1302–1309.
OECD. 2007. Understanding the Brain: The Birth of a Learning Science. Paris: Centre
for Educational Research and Innovation.
Ozcelik, E., N. E. Cagiltay, and N. S. Ozcelik. 2013. “The Effect of Uncertainty on
Learning in Game-Like Environments.”Computers & Education 67: 12–20.
Perrotta, C., G. Featherstone, H. Aston, and E. Houghton. 2013. Game-Based
Learning: Latest Evidence and Future Directions. Slough: NFER.
Richards, J., L. Stebbins, and K. Moellering. 2013. Games for a Digital Age: K-12
Market Map and Investment Analysis. New York: The Joan Ganz Cooney Center
at Sesame Workshop.
Royal Society, 2011. Brain Waves 2: Neuroscience Implications for Education and
Lifelong Learning. London: Royal Society.
Schultz, W. 1998. “Predictive Reward Signal of Dopamine Neurons.”Journal of
Neurophysiology 80 (1): 1–27.
Shizgal, P., and A. Arvanitogiannis. 2003. “Neuroscience: Gambling on Dopamine.”
Science 299 (5614): 1856–1858.
Wastiau, P., C. Kearney, and W. Van den Berghe. 2009. How Are Digital Games Used
in Schools? Complete Results of the Study. Final Report. Brussels: European
Schoolnet, EUN.
Weinstein, A. M. 2010. “Computer and Video Game Addiction –A Comparison
Between Game Users and Non-game Users.”The American Journal of Drug and
Alcohol Abuse 36 (5): 268–276.
Weisberg, D. S., F. C. Keil, J. Goodstein, E. Rawson, and J. R. Gray. 2008. “The
Seductive Allure of Neuroscience Explanations.”Journal of Cognitive
Neuroscience 20 (3): 470–477.
Whitton, N. 2011. “Game Engagement Theory and Adult Learning.”Simulation &
Gaming 42 (5): 596–609.
Whitton, N. 2014. Digital Games and Learning: Research and Theory. London:
Routledge.
20 P. Howard-Jones et al.
Downloaded by [University of Bristol] at 08:43 09 September 2014
