Evaluating outdoor play for children: virtual vs. tangible game objects in pervasive games.
ABSTRACT In this paper we report a case study where two versions of the same outdoor pervasive game were compared: one featuring a virtual game object and the other with a tangible representation of it. Our aim was to explore the effect on social interaction and physical activity; two characteristics of Head-Up Games. Based on evaluation with 27 children we can conclude that both approaches support Head-Up Games well, and offer different design opportunities that should be explored further. Categories and Subject Descriptors H.1.2 (Information Systems): User/Machine Systems.
- SourceAvailable from: Panos Markopoulos[Show abstract] [Hide abstract]
ABSTRACT: This paper reflects on the design process of games that are played by multiple players, involving high pace activity and embodied interaction. More specifically it argues that user testing with low fidelity prototypes, which is recommended in mainstream literature on methodology in the fields of human computer interaction and game design, is not appropriate when designing these kind of games. Designers should instead, as early as possible in the design process, experiment with technology and expose working prototypes to play test with children. A case study, in which we designed several games and tested in three iterations, is also presented. The games were designed for and tested with RaPIDO, a specially designed platform for prototyping mobile and interactive technology. Finally, we argue that our hypothesis regarding technology-rich prototyping is confirmed, since the feedback from the children concerned the realized interaction, and aspects of play and social interaction were experienced in real context, instead of an imagined way as a mock-up would have allowed.Proceedings of the 12th International Conference on Interaction Design and Children; 01/2013
- [Show abstract] [Hide abstract]
ABSTRACT: This paper explores how different interfaces to a problem-solving task affect how users perform it. Specifically, it focuses on a customized version of the game of Four-in-a-row and compares play on a physical, tangible game board with that conducted in mouse and touch-screen driven virtual versions. This is achieved through a repeated measures study involving a total of 36 participants and which explicitly assesses aspects of cognitive work through measures of time task, subjective workload, the projection of mental constructs onto external structures and the occurrence of explanatory epistemic actions. The results highlight the relevance of projection and epistemic action to this problem-solving task and suggest that the different interface forms afford instantiation of these activities in different ways. The tangible version of the system supports the most rapid execution of these actions and future work on this topic should explore the unique advantages of tangible interfaces in supporting epistemic actions.Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction; 02/2013
- [Show abstract] [Hide abstract]
ABSTRACT: There is a growing body of research in pervasive outdoor gaming, mainly focused on adult players playing games on smart phones. Published evaluations of the player experience in such games are largely based on anecdotal descriptions and post-play surveys. The latter approach is especially challenging to apply when the play test participants are children. Observations of game play so far have been ad hoc relying on unstructured observation, which makes it difficult to extract reliable conclusions from observations and to draw comparisons between different games. In this paper we present two methods developed specifically for evaluating the player experience in children’s outdoor games: the Outdoor Play Observation Scheme (OPOS) and GroupSorter. We discuss their application in three different case studies and conclude that OPOS is useful in quantifying the different types of play behavior in outdoor games; GroupSorter adds qualitative data on the play experience. Moreover, the application of GroupSorter is not limited to game development but can be used for obtaining user input in other context as well.Entertainment Computing 02/2013; 4(1):25-38.
IDC 2009 – Short Papers 3-5 June, 2009 – Como, Italy
Evaluating Outdoor Play for Children: Virtual vs. Tangible
Game Objects in Pervasive Games
Iris Soute, Maurits Kaptein, Panos Markopoulos
Department of Industrial Design
Technische Universiteit Eindhoven
P.O. Box 513, 5600 MB Eindhoven, The Netherlands
In this paper we report a case study where two versions of the
same outdoor pervasive game were compared: one featuring a
virtual game object and the other with a tangible representation of
it. Our aim was to explore the effect on social interaction and
physical activity; two characteristics of Head-Up Games. Based
on evaluation with 27 children we can conclude that both
approaches support Head-Up Games well, and offer different
design opportunities that should be explored further.
Categories and Subject Descriptors
H.1.2 [Information Systems]: User/Machine Systems.
Design, Experimentation, Human Factors.
Children, pervasive games, outdoor games
In  Soute and Markopoulos propose the concept of Head-Up
Games (HUGs): outdoor pervasive games for children that
support play patterns found in traditional game play. In contrast to
current pervasive games that mainly use PDA’s as gaming
interfaces (e.g. [4,5,6]), Soute and Markopoulos argue that
embedded gaming technologies can be designed without the need
to attend to a screen and can fit seamlessly into play encouraging
social engagement and physical activity.
Two possible approaches for representing a game object are a
tangible and a virtual interface. An intuitive assumption is that
tangible interfaces are engaging and enjoyable for children, since
they provide a natural way of interaction . In contrast, a
virtual game element cannot be ‘seen’, but in dynamics is more
similar to a computer game, which children also enjoy .
In this paper we describe the evaluation of a HUG in which a
tangible and a virtual game object are compared to investigate the
effect on social interaction and physical activity, and thus to see
which type supports HUGs best. For evaluating the amount of
social interaction and physical activity, the Outdoor Play
Observation Scheme (OPOS)  was used.
2. OBSERVATION SCHEME OPOS
OPOS was especially designed to evaluate outdoor games and
defines four different classes of behavior: physical activity –
distinguishes intensive, non-intensive and no physical activity;
focus – deals with what the players are looking at: each other,
game objects or something else; social interaction – codes the
types of interaction with other players, such as functional
(required to play the game), and non-functional interactions; and
general – this class is not related to play behaviors but is meant
for coding practical issues such as whether or not players are in
3. THE GAME “SAVE THE SAFE”
We evaluated two versions of a Head-Up Game called “Save the
Safe”. This game is a refinement of the game “Stop the Bomb” by
Hendrix et al. . The original game concept had been only partly
implemented. Given the successful evaluation of the game
concept, we decided to continue development and re-evaluate the
game. To avoid appealing to the violent fantasy element in the
original game, it was renamed to “Save the Safe” and the
narrative was adapted accordingly.
3.1 Game Play
At the start of the game players are randomly divided into two
teams of 4 players each: the burglars and the guards. The game
revolves around a safe and its key; the guards win the game when
they successfully guard the key from the burglars for the duration
of the game (5 minutes). The burglars win the game when they
steal the key from the guards and unlock the safe.
Each player wears a belt (see Figure 1). The belt contains LEDs to
indicate the player’s team color. Furthermore it contains a
vibration motor and an RF communication unit (Crossbow MICA
motes ). At startup the motes automatically form a network that
allows for easy communication between motes. Incidentally, the
communication unit is not meant for communication between
players, but is used for determining distances between players, by
measuring the signal strength of nearby belts.
The game can be played in two modes: a virtual and tangible
mode. In the virtual mode the key is represented by the vibration
motor in the belt. If a player possesses the key, his or her belt will
start vibrating. If another player comes sufficiently close, the key
will automatically be transferred to that player’s belt. In other
words the output appeals to the haptic sense, and there is no
tangible game object as such.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
IDC 2009, June 3–5, 2009, Como, Italy
Copyright 2009 ACM 978-1-60558-395-2/09/06… $5.00.
IDC 2009 – Short Papers 3-5 June, 2009 – Como, Italy
In the tangible mode, the key is represented by a physical ball.
Players can pass the key around by throwing the ball to each
other, and the burglars can steal it from the guards by intercepting
the ball. The belts are used in this game only for indicating the
Figure 1. Players chasing each other; inset: belt.
In the virtual mode, the interaction differs in several ways. No
physical manipulation is required to obtain the key – just being in
the vicinity of the key holder for approx. 3 seconds is enough. In
the virtual mode the only player who is informed of the location
of the key is the player who possesses it. In contrast, in the
tangible mode all players see the transfer of the key because they
can see the ball.
The safe is equipped with the same communication technology as
the belts. In the virtual mode, when a burglar in possession of the
key comes close to the safe, the key will be transferred to the safe,
which ends the game. In the tangible mode the burglars ’open’ the
safe by touching the safe with the ball.
27 children (15 girls, 12 boys) between 8 and 9 years old from a
primary school in the Netherlands participated in the evaluation.
As each game session required 8 children, some children
participated in two game sessions. Boys and girls were evenly
mixed over the sessions.
4.2 Procedure and Setup
All games were played outside on a school playing field. Two
video cameras were placed at two corners of the field.
In total, 4 sessions were held and in each session 4 games were
played: a training game, followed by both a game in the virtual
mode as well as a game in the tangible mode, and finally a game
in the mode of the participants’ choice. The order of the tangible
and virtual games was controlled for.
After all games were played, participants were taken inside to fill
in a short questionnaire. They were also asked to rank 8 elements
of the game, from ’most fun’ to ’least fun’. Finally, the children
were asked to repeat this task, however now as a group and we
recorded the ensuing discussion for analysis. A moderator
facilitated the conversation.
The footage of the cameras was annotated for each participant
with OPOS using Noldus Observer . A second coder coded
25% of the data, to calculate inter-rater reliability. This resulted in
K = 0.53 for the category physical activity and K = 0.43 for focus.
According to  these values indicate moderate agreement
between coders. For social interaction K = -0.24, indicating poor
As each game had a different duration, the results have been
computed to an average percentage of the total time for the
classes ’physical activity’ and ’focus’. For the class ’social
interaction’ results have been computed to average number of
events per minute.
Figure 2. Results physical activity, with 95% confidence
A 2 X 2 mixed subject analysis of variance using game type
(tangible vs. virtual) and game order (tangible first vs. virtual
first) as independent variables and the average percentage of time
on different types of activities for each child was performed.
When analyzing the overall physical activity (intensive and non-
intensive) it was clear that there was no main effect of game type,
F(1,22) = 0.884, p = 0.357 and no main effect of game order,
F(1,22) = 2.613, p = 0.120. Furthermore, no significant interaction
was found, F(1,22) = 0.051, p = 0.822. Thus children are equally
active in the two game versions.
However, when analyzing the percentage of intensive physical
activity using a similar 2 X 2 mixed subject ANOVA it was clear
that there was a significant main effect of game type, F(1,22) =
12.278, p<0.05. No significant main effect of order, F(1,22) =
1.880, p = 0.184 and no significant interaction, F(1,22) = 0.137, p
= 0.715 were found. This leads us to conclude that in the virtual
mode, while in both games children are overall equally active,
children were more often intensely active than in the tangible
From Figure 3 we can immediately see that there is a large
difference in focus on ’looking at game objects’ and ’looking at
other players’. Using a similar 2 X 2 ANOVA it was clear that
children focused significantly more often on game objects in the
tangible mode than in the virtual mode, F(1,22)=131.521, p<0.05.
No significant effect was found for order, F(1,22)=1.546,
p=0.227, nor was the interaction effect significant, F(1,22) =
3.186, p = 0.088. The most likely reason for this is that in the
virtual mode there are less game objects to look at. Since players
look at each other significantly more often in the virtual mode,
one could expect that this should lead to more social interaction;
however this is not supported by the social interaction data.
IDC 2009 – Short Papers 3-5 June, 2009 – Como, Italy
Figure 3. Results focus, with 95% confidence intervals.
There are no significant differences in social interaction per
minute between the two game modes, except for ’interaction with
non-players’. In the virtual mode there was more interaction with
non-players; players asked questions or talked to the
experimenter. Outcomes of the results of a 2 X 2 mixed subject
ANOVA on the social interaction scores are presented in Table 1.
Table 1. Results for social interaction.
Type Order Interaction
F(1,22) p F(1,22) p F(1,22) p
0.536 0.472 0.249 0.623 0.021 0.886
4.789 0.040* 1.535 0.228 2.702 0.114
1.565 0.224 0.634 0.434 8.062 0.010
* significant at 5% level
a too few observations
5.2 Again, again
The participants were allowed to choose which game mode they
wanted to play in the final game. This is an adaptation of the
'again, again' method as proposed by Read and MacFarlane ,
and can be interpreted as a measure of fun. In three out of four
sessions the virtual mode was preferred over the tangible mode.
5.3 Preference and Rankings
In a short questionnaire, we asked the children 1) which game
mode they thought was most fun, 2) in which mode the physical
interaction was most fun, and 3) in which mode the social
interaction was most fun. The results are shown in Table 2.
Table 2: Which mode was most fun.
Most fun Virtual mode Tangible mode
Overall 23 4
Physical activity* 19 7
Social interaction 19 8
* one participant did not record his preference
Table 3. Ranking of game elements (1=most fun, 8 = least fun).
Item name Individual ranking Group ranking
Belt 1 2
Running 2 3
Playing together 3 1
Which team 4 6
Ball 5 5
Winning 6 8
Safe 7 7
Discussing 8 4
The participants were asked to rank eight game elements from
’most fun’ to ’least fun’, both individually as well as group-wise.
From the rankings a combined rank can be calculated; the
combined ranks are shown in Table 3..
5.4 Group Discussion
Children’s comments during the group ranking were analyzed,
using conventional content analysis , i.e. the coding scheme
was built up during coding of the transcripts.
Many references were made to the belt and its impact on game
play: “Nobody knows who possesses it [the key]”, “It makes you
run more”, “I like the vibration. You don’t see that in other
games”, and “You could not easily take it [the key] away from
Concerning the ball (the tangible key) children mentioned “the
ball is more fun, since it makes you play together more”,
“Everybody can see that you have it [the ball]”, “With the ball
you have to stand still [to throw it], with the belt you can keep
Children found it easier to discuss with each other during the
game with the tangible element: “It’s easier with the ball, since
then you can stand still and you can discuss. If you move and run
it’s a little hard.”
6. DISCUSSION AND CONCLUSION
Regarding the coding of the videos we found that the inter-rater
reliability for social interaction was poor. The reason for this is
that it was quite difficult to judge children’s verbal utterances
from the videos. A possible improvement would be to equip
participants with small microphones, so each participant’s speech
is recorded separately.
From rankings, again-again and the group discussion, we
conclude that children preferred the virtual game. We attribute
this to the new technology introduced to the children, as many
positive comments were made concerning the haptic feedback of
the belt; children commented that it was new to them and that
they enjoyed it very much.
The merits of tangible interaction have been argued manifold; but
empirical research so far has produced limited evidence to
substantiate these claims . Still, the reader is warned not to
conclude prematurely the case against tangibles. For one, many
different definitions of tangibility exist : both the belt and the
ball could for some be considered as tangibles. The distinction
between the two is that the ball is manipulated as a physical
object while the belt provides a haptic output to the player.
Children mentioned during the discussion how the direct
manipulation of the ball was easier for them than the indirect
manipulation of the key in virtual mode. This is in line with what
Xie et al.  found and with what one would expect regarding
the intuitiveness and usability of tangible objects.
Furthermore, the very physicality of the ball changes the game
play dramatically. To ensure construct validity in a comparison
between a tangible and a non tangible version of a game, the two
should be identical in all ways apart from the physical
manipulation of the artifact. In this evaluation, the visibility of the
game object was unavoidably linked with its physicality: players
could not see who had the key in virtual mode, while they could
all see who had the ball in tangible mode. A straight comparison
IDC 2009 – Short Papers 3-5 June, 2009 – Como, Italy
could be obtained if, for example, a light would indicate whether
a player has the key or not in the case of the virtual version of the
game. While such a choice could have lent higher construct
validity to the conclusions, it would be at the expense of external
validity. In designing a game, we have to make use of the
advantages of each medium to design an enjoyable experience.
Equalizing the games in all other aspects but the physicality of the
key, would lead to losing the added suspense of discovering or
concealing who is in possession of the virtual key.
Given the above one can conclude the following; head up games
can be supported well both by using physical as well as virtual
game objects. These two approaches offer different design
opportunities; designers and researchers should not assume it as
self evident that physicality is the appropriate approach to
interaction design. Moreover, the relative advantages to both need
to be explored in different design contexts. In the specific context
of game design we saw that play resembled more traditional sport
games when a ball was used; there is perhaps more opportunity to
innovate and explore new experiences when virtual game objects
are used. Further research is needed to obtain a more refined
account of how physicality and virtuality of game objects impact
social interaction, physical activity and enjoyment of the game.
The research is supported by the European Community under the
IST program (FP6-2004-IST-4) Project PASION. The authors
would like to thank Koen Hendrix and Jeanine Kierkels for their
help during the evaluation of the game. Furthermore, the authors
thank all children, teachers and their school for their cooperation.
 Crossbow Technology. http://www.xbow.com/.
 Noldus Information Technology. http://www.noldus.com/.
 Bakker, S., Markopoulos, P., and Kort, Y.D. OPOS: an
observation scheme for evaluating head-up play. Proc. of
NordiCHI '08, ACM (2008), 33-42.
 Benford, S., Rowland, D., Flintham, M., et al. Life on the
edge: supporting collaboration in location-based
experiences. Proc. of CHI '05, ACM (2005), 721-730.
 Björk, S. and Ljungstrand, P. Pirates! In Space Time Play.
 Cheok, A., Anuroop Sreekumar, Lei, C., and Thang, L.
Capture the flag: mixed-reality social gaming with smart
phones. Pervasive Computing, IEEE 5, 2 (2006), 62-69.
 Hendrix, K., Yang, G., Mortel, D.V.D., Tijs, T., and
Markopoulos, P. Designing a Head-Up Game for Children.
Proc. of British HCI '08, (2008).
 Hornecker, E. and Buur, J. Getting a grip on tangible
interaction: a framework on physical space and social
interaction. Proc. of CHI'06, ACM (2006), 437-446.
 Hsieh, H. and Shannon, S.E. Three Approaches to
Qualitative Content Analysis. Qual Health Res 15, 9 (2005),
 Landis, J.R. and Koch, G.G. The measurement of observer
agreement for categorical data. Biometrics 33, 1 (1977), 159-
 Marshall, P. Do tangible interfaces enhance learning? Proc.
of TEI '07, ACM (2007), 163-170.
 Read, J.C. and MacFarlane, S. Using the fun toolkit and
other survey methods to gather opinions in child computer
interaction. Proc. of IDC '06, ACM (2006), 81-88.
 Soute, I. and Markopoulos, P. Head Up Games: The Games
of the Future Will Look More Like the Games of the Past. In
Human-Computer Interaction – INTERACT 2007. 2008,
 Verenikina, I., Harris, P., and Lysaght, P. Child's play:
computer games, theories of play and children's
development. Proc. of the IFIP WG 3.5 open conference on
Young children and learning technologies, Australian
Computer Society, Inc. (2003), 99-106.
 Xie, L., Antle, A.N., and Motamedi, N. Are tangibles more
fun?: comparing children's enjoyment and engagement using
physical, graphical and tangible user interfaces. Proc. of TEI
'08, ACM (2008), 191-198.