BookPDF Available

Human Computer Confluence: Transforming Human Experience Through Symbiotic Technologies

Authors:

Abstract

Human-computer confluence refers to an invisible, implicit, embodied or even implanted interaction between humans and system components. New classes of user interfaces are emerging that make use of several sensors and are able to adapt their physical properties to the current situational context of users. A key aspect of human-computer confluence is its potential for transforming human experience in the sense of bending, breaking and blending the barriers between the real, the virtual and the augmented, to allow users to experience their body and their world in new ways. Research on Presence, Embodiment and Brain-Computer Interface is already exploring these boundaries and asking questions such as: Can we seamlessly move between the virtual and the real? Can we assimilate fundamentally new senses through confluence? The aim of this book is to explore the boundaries and intersections of the multidisciplinary field of HCC and discuss its potential applications in different domains, including healthcare, education, training and even arts.
Andrea Gaggioli, Alois Ferscha, Giuseppe Riva, Stephen Dunne, Isabelle Viaud-
Delmon
Human Computer Confluence
Transforming Human Experience Through Symbiotic Technologies
Andrea Gaggioli, Alois Ferscha, Giuseppe Riva,
Stephen Dunne, Isabelle Viaud-Delmon
Human Computer
Confluence
Transforming Human Experience through Symbiotic
Technologies
Managing Editor: Aneta Przepiórka
Associate Editor: Pietro Cipresso
Language Editor: Catherine Lau
Published by De Gruyter Open Ltd, Warsaw/Berlin
Part of Walter de Gruyter GmbH, Berlin/Munich/Boston
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 license,
which means that the text may be used for non-commercial purposes, provided credit is given to the
author. For details go to http://creativecommons.org/licenses/by-nc-nd/3.0/.
© 2016 Andrea Gaggioli, Alois Ferscha, Giuseppe Riva, Stephen Dunne, Isabelle Viaud-Delmon
and chapters’ contributors
eBook (PDF) ISBN 978-3-11-047113-7
eBook (EPUB) ISBN 978-3-11-047169-4
Hardcover ISBN 978-3-11-047112-0
Bibliographic information published by the Deutsche Nationalbibliothek.
The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed
bibliographic data are available in the Internet at http://dnb.dnb.de.
Managing Editor: Aneta Przepiórka
Associate Editor: Pietro Cipresso
Language Editor: Catherine Lau
www.degruyteropen.com
Cover illustration: © Thinkstock ra2studio
Complimentary copy, not for sale.
Contents
List of contributing authors   XIII
Foreword 1
References 4
Section I: Conceptual Frameworks and Models 5
Alois Ferscha
1 A Research Agenda for Human Computer Confluence   7
1.1 Introduction 8
1.2 Generations of Pervasive / Ubiquitous (P/U) ICT 9
1.3 Beyond P/U ICT: Socio-Technical Fabric 11
1.4 Human Computer Confluence (HCC) 12
1.5 The HCC Research Agenda 13
1.5.1 Large Scale Socio-Technical Systems 13
1.5.2 Ethics and Value Sensitive Design 14
1.5.3 Augmenting Human Perception and Cognition 14
1.5.4 Empathy and Emotion 15
1.5.5 Experience and Sharing 15
1.5.6 Disappearing Interfaces 15
1.6 Conclusion 16
References 16
David Benyon and Oli Mival
2 Designing Blended Spaces for Collaboration   18
2.1 Introduction 18
2.2 Blending Theory 21
2.3 Blended Spaces 24
2.4 Designing the ICE 26
2.4.1 The physical Space 26
2.4.2 The Digital Space 27
2.4.3 The Conceptual Space 28
2.4.4 The Blended Space 29
2.5 The TACIT Framework 29
2.5.1 Territoriality 30
2.5.2 Awareness 31
2.5.3 Control 32
2.5.4 Interaction 33
2.5.5 Transitions   34
2.6 Discussion 35
2.6.1 Designing the Physical Space 35
2.6.2 Designing the Digital Space 36
2.6.3 Designing the Conceptual Space 36
2.6.4 Designing the Blended Space 36
2.7 Conclusions 37
References 38
Francesca Morganti
3 “Being There” in a Virtual World: an Enactive Perspective on Presence and
its Implications for Neuropsychological Assessment and Rehabilitation 
 40
3.1 Introduction 40
3.2 Enactive Cognition 42
3.3 Enactive Neuroscience 44
3.4 Enactive Presence in Virtual Reality 45
3.5 Enactive Technologies in Neuropsychology 48
3.6 Enactive Knowledge and Human Computer Confluence 50
References 53
Giuseppe Riva
4 Embodied Medicine: What Human-Computer Confluence Can Offer to Health
Care   55
4.1 Introduction 55
4.2 Bodily Self-Consciousness 57
4.2.1 Bodily Self-Consciousness: its Role and Development 58
4.2.2 First-Person Spatial Images: the Common Code of Bodily Self-
Consciousness 63
4.3 The Impact of Altered Body Self-Consciousness on Health Care 67
4.4 The Use of Technology to Modify Our Bodily Self-Consciousness 70
4.5 Conclusions 73
References 75
Bruno Herbelin, Roy Salomon, Andrea Serino and Olaf Blanke
5 Neural Mechanisms of Bodily Self-Consciousness and the Experience of
Presence in Virtual Reality   80
5.1 Introduction 80
5.2 Tele-Presence, Cybernetics, and Out-of-Body Experience 81
5.3 Immersion, Presence, Sensorimotor Contingencies, and Self-
Consciousness 83
5.4 Presence and Bodily Self-Consciousness 85
5.4.1 Agency 86
5.4.2 Body Ownership and Self-Location 88
5.5 Conclusion 91
References 92
Andrea Gaggioli
6 Transformative Experience Design   97
6.1 Introduction 97
6.2 Transformation is Different From Gradual Change 98
6.2.1 Transformative Experiences Have an Epistemic Dimension and a
Personal Dimension 101
6.2.2 Transformative Experience as Emergent Phenomenon 105
6.2.3 Principles of Transformative Experience Design 106
6.2.3.1 The Transformative Medium 106
6.2.3.1.1 I Am a Different Me: Altering Bodily Self-Consciousness 107
6.2.3.1.2 I Am Another You: Embodying The Other 108
6.2.3.1.3 I Am in a Paradoxical Reality: Altering the Laws of Logic 109
6.2.3.2 Transformative Content 111
6.2.3.2.1 Emotional Affordances 111
6.2.3.2.2 Epistemic Affordances 112
6.2.3.3 The Transformative Form 113
6.2.3.3.1 Cinematic Codes 113
6.2.3.3.2 Narratives 113
6.2.3.4 The Transformative Purpose 115
6.2.3.4.1 Transformation as Liminality 115
6.2.3.4.2 The Journey Matters, Not the Destination 116
6.3 Conclusion: the Hallmarks of Transformative Experience Design 117
References 118
Section II: Emerging Interaction Paradigms 123
Frédéric Bevilacqua and Norbert Schnell
7 From Musical Interfaces to Musical Interactions   125
7.1 Introduction 125
7.2 Background 127
7.3 Designing Musical Interactions 128
7.4 Objects, Sounds and Instruments 129
7.5 Motion-Sound Relationships   132
7.5.1 From Sound to Motion 132
7.5.2 From Motion to Sound 133
7.6 Towards Computer-Mediated Collaborative Musical Interactions 134
7.7 Concluding Remarks 136
References 136
Fivos Maniatakos
8 Designing Three-Party Interactions for Music Improvisation Systems 
 141
8.1 Introduction 141
8.2 Interactive Improvisation Systems 143
8.2.1 Compositional Trends and Early Years 143
8.2.2 Interacting Live With the Computer 144
8.3 Three Party Interaction in Improvisation 145
8.4 The GrAIPE Improvisation System 148
8.5 General Architecture for Three Party Improvisation in Collective
Improvisation 150
8.6 Interaction Modes 152
8.6.1 GrAIPE as an Instrument 152
8.6.2 GrAIPE as a Player 154
8.7 Conclusion 154
References 155
Doron Friedman and Béatrice S. Hasler
9 The BEAMING Proxy: Towards Virtual Clones for Communication   156
9.1 Introduction 156
9.2 Background 157
9.2.1 Semi-Autonomous Virtual Agents 157
9.2.2 Digital Extensions of Ourselves 158
9.2.3 Semi-Autonomous Scenarios in Robotic Telepresence 159
9.3 Concept and Implications 160
9.3.1 Virtual Clones 160
9.3.2 Modes of Operation 162
9.3.3 “Better Than Being You” 165
9.3.4 Ethical Considerations 166
9.4 Evaluation Scenarios 167
9.4.1 The Dual Gender Proxy Scenario 168
9.4.2 Results 171
9.5 Conclusions   171
References 172
Robert Leeb, Ricardo Chavarriaga, and José d. R. Millán
10 Brain-Machine Symbiosis   175
10.1 Introduction 175
10.2 Applied Principles 178
10.2.1 Brain-Computer Interface Principle 178
10.2.2 The Context Awareness Principle 178
10.2.3 Hybrid Principle 180
10.3 Direct Brain-Controlled Devices 181
10.3.1 Brain-Controlled Wheelchair 181
10.3.2 Tele-Presence Robot Controlled by Motor-Disable People 183
10.3.3 Grasp Restoration for Spinal Cord Injured Patients 185
10.4 Cognitive Signals and Mental States 187
10.4.1 Error-Related Potentials 187
10.4.2 Decoding Movement Intention 188
10.4.3 Correlates of Visual Recognition and Attention 189
10.5 Discussion and Conclusion to Chapter 10 190
References 192
Jonathan Freeman, Andrea Miotto, Jane Lessiter, Paul Verschure, Pedro Omedas, Anil
K. Seth, Georgios Th. Papadopoulos, Andrea Caria, Elisabeth André, Marc Cavazza,
Luciano Gamberini, Anna Spagnolli, Jürgen Jost, Sid Kouider, Barnabás Takács,
Alberto Sanfeliu, Danilo De Rossi, Claudio Cenedese, John L. Bintliff, and Giulio
Jacucci
11 The Human as the Mind in the Machine: Addressing Big Data   198
11.1 ‘Implicitly’ Processing Complex and Rich Information 198
11.2 Exploiting Implicit Processing in the Era of Big Data: the CEEDs Project
 201
11.3 A Unified High Level Conceptualisation of CEEDs 206
11.4 Conclusion 209
References 210
Alessandro D’Ausilio, Katrin Lohan, Leonardo Badino and Alessandra Sciutti
12 Studying Human-Human interaction to build the future of Human-Robot
interaction   213
12.1 Sensorimotor Communication 213
12.1.1 Computational Advantages of Sensorimotor Communication 214
12.2 Human-to-Human Interaction   215
12.2.1 Ecological Measurement of Human-to-Human Information Flow 215
12.3 Robot-to-Human Interaction 216
12.3.1 Two Examples on How to Build Robot-to-Human Information Flow
 218
12.4 Evaluating Human-to-Robot Interaction 219
12.4.1 Short term Human-to-Robot Interaction 220
12.4.2 Long Term Human-to-Robot Interaction 221
12.5 Conclusion 222
References 223
Section III: Applications 227
Joris Favié, Vanessa Vakili, Willem-Paul Brinkman, Nexhmedin Morina and Mark A.
Neerincx
13 State of the Art in Technology-Supported Resilience Training For Military
Professionals   229
13.1 Introduction 229
13.2 Psychology 230
13.2.1 Resilience 230
13.2.2 Hardiness 231
13.3 Measuring Resilience 231
13.3.1 Biomarkers 232
13.3.2 Emotional Stroop Test 233
13.3.3 Startle Reflex 233
13.3.4 Content Analysis of Narratives 235
13.4 Resilience Training 235
13.4.1 Stress Inoculation Training (SIT) 235
13.4.2 Biofeedback Training 235
13.4.3 Cognitive Reappraisal 236
13.5 Review of Resilience Training Systems 237
13.5.1 Predeployment Stress Inoculation Training (Presit) & Multimedia
Stressor Environment (Mse) 237
13.5.2 Stress Resilience in Virtual Environments (Strive) 237
13.5.3 Immersion and Practice of Arousal Control Training (Impact) 238
13.5.4 Personal Health Intervention Tool (PHIT) for Duty 238
13.5.5 Stress Resilience Training System 239
13.5.6 Physiology-Driven Adaptive Virtual Reality Stimulation for Prevention
and Treatment of Stress Related Disorders 239
13.6 Conclusions 239
References 240
Mónica S. Cameirão and Sergi Bermúdez i Badia
14 An Integrative Framework for Tailoring Virtual Reality Based Motor
Rehabilitation After Stroke   244
14.1 Introduction 244
14.2 Human Computer Confluence (HCC) in neurorehabilitation 245
14.3 The RehabNet Framework 249
14.4 Enabling VR Neurorehabilitation Through its Interfaces 250
14.4.1 Low Cost at Home VR Neurorehabilitation 250
14.4.2 Enabling VR Neurorehabilitation Through Assistive Robotic Interfaces
 251
14.4.3 Closing the Act-Sense Loop Through Brain Computer Interfaces 252
14.5 Creating Virtual Reality Environments For Neurorehabilitation 253
14.6 Looking Forward: Tailoring Rehabilitation Through Neuro-Computational
Modelling 257
References 258
Pedro Gamito, Jorge Oliveira, Rodrigo Brito, and Diogo Morais
15 Active Confluence: A Proposal to Integrate Social and Health Support with
Technological Tools   262
15.1 Introduction 262
15.1.1 An Ageing Society 263
15.1.2 Ageing and General and Mental Health 263
15.1.3 Social Support and General and Mental Health 265
15.1.4 Ageing and Social Support 265
15.2 Active Confluence: A Case for Usage 266
15.2.1 Perception and Interaction 267
15.2.2 Sensing 268
15.2.3 Physical and Cognitive Multiplayer Exercises 269
15.2.4 Integrated Solutions for Active Ageing: The Vision 270
15.3 Conclusion 271
References 272
Georgios Papamakarios, Dimitris Giakoumis, Manolis Vasileiadis, Anastasios Drosou
and Dimitrios Tzovaras
16 Human Computer Confluence in the Smart Home Paradigm: Detecting
Human States and Behaviours for 24/7 Support of Mild-Cognitive
Impairments   275
16.1 Introduction 275
16.2 Detecting Human States and Behaviours at Home 277
16.2.1 Ambient Sensor-Based Activity Detection   277
16.2.2 Resident Movement-Based Activity Detection 278
16.2.3 Applications and Challenges of ADL Detection in Smart Homes 278
16.3 A Behaviour Monitoring System for 24/7 Support of MCI 279
16.3.1 Activity Detection Based on Resident Trajectories 279
16.3.2 Ambient Sensor-Based Activity Detection 281
16.3.3 Activity Detection Based on Both Sensors and Trajectories 284
16.4 Experimental Evaluation 284
16.4.1 Data Collection 284
16.4.2 Experimental Results 286
16.4.2.1 Kitchen Experiment 287
16.4.2.2 Apartment Experiment 287
16.5 Toward Behavioural Modelling and Abnormality Detection 289
16.6 Conclusion 290
References 291
Andreas Riener, Myounghoon Jeon and Alois Ferscha
17 Human-Car Confluence: “Socially-Inspired Driving Mechanisms”   294
17.1 Introduction 294
17.2 Psychology of Driving and Car Use 295
17.2.1 From Car Ownership to Car Sharing 295
17.2.2 Intelligent Services to Support the Next Generation Traffic 296
17.3 Human-Car Entanglement: Driver-Centered Technology and Interaction
 298
17.3.1 Intelligent Transportation Systems (ITS) 298
17.3.2 Recommendations 299
17.4 Socially-Inspired Traffic and Collective-Aware Systems 300
17.4.1 The Driver: Unpredictable Behavior 302
17.4.2 The Car: Socially Ignorant 303
17.4.3 Networks of Drivers and Cars: Socially-Aware? 305
17.4.4 Recommendations 305
17.4.5 Experience Sharing: A Steering Recommender System 306
17.5 Conclusion 307
References 307
List of Figures   311
List of Tables   317
Alois Ferscha
Johannes Kepler University, Linz, Austia
Chapter 1
David Benyon
Edinburgh Napier University, UK
Chapter 2
Oli Mival
Edinburgh Napier University, UK
Chapter 2
Francesca Morganti
University of Bergamo, Italy
Chapter 3
Giuseppe Riva
Catholic University of Sacred Heart, Milan, Italy
Istituto Auxologico Italiano, Milan, Italy
Chapter 4
Bruno Herbelin
Ecole Polytechnique Fédérale de Lausanne,
Switzerland
Chapter 5
Roy Salomon
Ecole Polytechnique Fédérale de Lausanne,
Switzerland
Chapter 5
Andrea Serino
Università di Bologna, Italy
Ecole Polytechnique Fédérale de Lausanne,
Switzerland
Chapter 5
Olaf Blanke
Ecole Polytechnique Fédérale de Lausanne,
Switzerland
University Hospital, Geneva, Switzerland
Chapter 5
Andrea Gaggioli
Catholic University of Sacred Heart, Milan, Italy
Istituto Auxologico Italiano, Milan, Italy
Chapter 6
Frederic Bevilacqua
Ircam, CNRS, UPMC, Paris, France
Chapter 7
Norbert Schnell
Ircam, CNRS, UPMC, Paris, France
Chapter 7
Fivos Maniatakos
IRCAM-CentrePompidou, Paris, France
Chapter 8
Doron Friedman
Sammy Ofer School of Communications of Inter-
disciplinary Center Herzliya, Herzliya, Israel
Chapter 9
Béatrice S. Hasler
Sammy Ofer School of Communications of Inter-
disciplinary Center Herzliya, Herzliya, Israel
Chapter 9
Robert Leeb
École Polytechnique Fédérale de Lausanne
Switzerland
Chapter 10
Ricardo Chavarriaga
École Polytechnique Fédérale de Lausanne
Switzerland
Chapter 10
José d. R. Millán
École Polytechnique Fédérale de Lausanne
Switzerland
Chapter 10
List of contributing authors
XIVList of contributing authors
Jonathan Freeman
Goldsmiths, University of London, UK
i2 media research ltd
Chapter 11
Andrea Miotto
Goldsmiths, University of London, UK
i2 media research ltd
Chapter 11
Jane Lessiter
Goldsmiths, University of London, UK
i2 media research ltd
Chapter 11
Paul Verschure
Universitat Pompeu Fabra, Barcelona, Spain
Institució Catalana de Recerca i Estudis Avançats
(ICREA), Barcelona, Spain
Chapter 11
Pedro Omedas
Universitat Pompeu Fabra, Barcelona, Spain
Chapter 11
Anil K. Seth
University of Sussex, Falmer, Brighton, UK
Chapter 11
Georgios Th. Papadopoulos
Centre for Research and Technology Hellas/
Information Technologies Institute (CERTH/ITI),
Greece
Chapter 11
Andrea Caria
Eberhard Karls Universität Tübingen, Germany
Chapter 11
Elisabeth André
Universität Augsburg, Germany
Chapter 11
Marc Cavazza
Teesside University, UK
Chapter 11
Luciano Gamberini
University of Padua, Italy
Chapter 11
Anna Spagnolli
University of Padua, Italy
Chapter 11
Jürgen Jost
Max Planck Institute for Mathematics in the Sci-
ences, Leipzig, Germany
Chapter 11
Sid Kouider
CNRS and Ecole Normale Supérieure, Paris,
France
Chapter 11
Barnabás Takács
Technical University of Budapest, Hungary
Chapter 11
Alberto Sanfeliu
Institut de Robotica i Informatica Industrial
(CSIC-UPC), Barcelona, Spain
Chapter 11
Danilo De Rossi
University of Pisa, Italy
Chapter 11
Claudio Cenedese
Electrolux Global Technology Center, Udine, Italy
Chapter 11
John L. Bintliff
Universiteit Leiden, the Netherlands
Chapter 11
Giulio Jacucci
University of Helsinki, Finland
Aalto University, Finland
Chapter 11
Alessandro D’Ausilio
IIT – Italian Institute of Technology, Genova, Italy
Chapter 12
List of contributing authors XV
Katrin Solveig Lohan
HWU-Heriot-Watt University, Genova, Italy
Chapter 12
Leonardo Badino
IIT – Italian Institute of Technology, Genova,
Italy
Chapter 12
Alessandra Sciutti
IIT – Italian Institute of Technology, Genova,
Italy
Chapter 12
Joris Favié
Delft University of Technology,The Netherlands
Chapter 13
Vanessa Vakili
Delft University of Technology,The Netherlands
Chapter 13
Willem-Paul Brinkman
Delft University of Technology,The Netherlands
Chapter 13
Nexhmedin Morina
University of Amsterdam, The Netherlands
Chapter 13
Mark A. Neerincx
Delft University of Technology,The Netherlands
TNO Human Factors, Soesterberg, The Nether-
lands
Chapter 13
Monica Cameirao
Universidade da Madeira, Portugal
Polo Científico e Tecnológico da Madeira,
Portugal
Chapter 14
Sergi Bermúdez i Badia
Universidade da Madeira, Portugal
Polo Científico e Tecnológico da Madeira,
Portugal
Chapter 14
Pedro Gamito
Lusophone University, Lisbon, Portugal
Chapter 15
Jorge Oliveira
Lusophone University, Lisbon, Portugal
Chapter 15
Rodrigo Brito
Lusophone University, Lisbon, Portugal
Chapter 15
Diogo Morais
Lusophone University, Lisbon, Portugal
Chapter 15
Dimitris Giakoumis
Information Technologies Institute, Thessa-
loniki, Greece
Chapter 16
Georgios Papamakarios
Information Technologies Institute, Thessa-
loniki, Greece
Chapter 16
Manolis Vasileiadis
Information Technologies Institute, Thessa-
loniki, Greece
Chapter 16
Anastasios Drosou
Information Technologies Institute, Thessa-
loniki, Greece
Chapter 16
XVIList of contributing authors
Dimitrios Tzovaras
Information Technologies Institute, Thessa-
loniki, Greece
Chapter 16
Myounghoon Jeon
Michigan Technological University, Houghton,
USA
Chapter 17
Andreas Riener
Johannes Kepler University Linz, Austria
Chapter 17
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License
© 2016 Walter Van de Velde
Walter Van de Velde
Foreword
The editors of this volume bring together a collection of papers that, taken together,
provide an interesting snapshot of current research on Human Computer Conflu-
ence (HCC). HCC stands for a particular vision of how computing devices and human
beings (users) can be fruitfully combined. The term was coined some 10 years ago in
an attempt to make sense of the increasingly complex technological landscape that
was emerging from different interrelated research strands in Information and Com-
munication Technology (ICT). It was clear that computers could no longer be seen just
as machines that execute algorithms for turning input into output data. This simply
seemed to miss a lot of the ways in which computers were starting to be used, in
which ‘users’ could interact with them and the multitudes of shapes they could take.
After the era of ‘desktop computing’, the power of an application was no longer just
residing in the sophistication of its algorithmic core, but also, or maybe even mainly,
in the way it blends into the user’s sensation of being engaged into some value-add-
ing experience (for work, for health, for entertainment,…). Moreover, everything that
had been learned about how to design good screen-based interfaces seemed useless
for dealing with settings of, for instance, wearable or embedded computing. In order
to capture all this there was a pressing need for new concepts, a new language, new
ways of measuring and comparing things.
Human Computer Confluence became the name of a ‘proactive initiative’ (EC,
2014), funded by the European Commission’s Future and Emerging Technologies
(FET) programme under the Seventh Framework Programme for Research (FP7). In a
nutshell, the mission of FET is to explore radically new technological paradigms that
can become the future assets for a globally competitive Europe that is worth to live in.
A FET proactive initiative aims to assemble a critical mass of research around such
a technological paradigm in order to shape its distinct research agenda and to help
building up the multi-disciplinary knowledge base from which the paradigm can be
explored and further developed into a genuinely new line of technology.
Not all the work that is documented in this volume has been funded through
FET. Many of its authors would not even say that they are doing their work under
the umbrella of HCC. HCC is not a strictly defined school of thought. It is rather an
undercurrent that permeates across many lines of contemporary research activities in
information and communication technologies. In that sense, the concept of Human
Computer Confluence is more like a road sign marking a direction in the socio/tech-
nological landscape, rather than a precise destination.
1The views expressed are those of the author and not necessarily those of the European Commission
2Foreword
What’s in a name? Why was there a need for a new term or concept such as
Human Computer Confluence? After all, there was already ‘ubiquitous computing’
to capture the visionary idea of anywhere, anytime and contextualised computing.
After all, there were already emerging areas like Presence research and Interac-
tion Design, both with their essential focus on techniques to develop characterise
and even measure the ‘user experience’, be it in virtual reality, augmented reality
or embedded and ambient computing/settings. These areas of research are all very
relevant for the emerging HCC research agenda, and it is at their intersection that the
idea of HCC takes shape.
As should be obvious from the name, perhaps the most essential feature of HCC is
that it is not just a technological research agenda. Somewhat retrospectively, one can
argue that its whole raison d’être was to counterbalance the technology-dominated
vision of ubiquitous computing that (with a detour through ambient intelligence)
is now developing into Internet of Things. The whole human side of it that was ini-
tially very strong (especially in Disappearing Computing and Ambient Intelligence)
got overpowered by the more pressing and concrete technological research agenda
of components, systems, protocols and infrastructures. With the more fundamental
questions of the human side of the equation temporarily side lined, there was an
obvious need for longer term research such as funded by FET to do some groundwork
on this. It is in this context that HCC both as a term and as an initiative was born.
Ultimately, it will be in its contribution to turning Ubiquity and Internet of Things
into something worth to have for Europe’s citizens that its long-term impact should
be measured.
It is worth mentioning the other big school of thought for a human/technology
combination, namely the convergence research agenda of NBIC – Nano-Bio-Info-
Cogno. This is a strongly reductionist vision of human engineerability, based on its
reduction to elementary material and ultimately engineerable building blocks over
which perfect control can be exercised. In its most radical versions it leads to Transhu-
manism where human nature itself gets inevitably replaced by a superior bio-techno
hybrid nature. Human Computer Confluence points to an alternative approach of
trying to understand and enhance the human sense and experience of technological
augmentation by building technology that takes into account the full human nature
of the ‘user’. In that sense it is also a good illustration of what the European version of
the NBIC convergence agenda could look like (HLEG, 2004).
HCC addresses what seems to be a paradox: how can we become more human
through technology, instead of less?²
Humans as equal part of the system to be studied: HCC cannot be reduced to
a purely technological challenge. Whereas before the human was a factor to be
2This list is based on the outcome of a panel discussion at the second HC2 Summer School, IRCAM,
2013 (HC2, 2013)
Foreword 3
taken into account (for example for usability) it is now an equally important part
of the study. This redefines the boundaries of the ‘system’, and hybridises the
nature of it in a fundamental way. The old idea of interface as the fine line that
cuts between system and user is no longer tenable. One rather has to think about
how complex process on both side interact and intertwine, at different levels (for
instance, physical, behavioural, cognitive) and based on a deep understanding
of both.
Empowerment rather than augmentation: HCC is not about augmentation of spe-
cific human capabilities (for example force, memory, concentration, perception,
reasoning), but about empowering humans to use their abilities in new ways,
themselves deciding when and what for.
Social enhancement rather than individual superiority: it appears useful to
anchor HCC in a networked view of computing, such as ubiquity or internet of
things, in order to keep it focused on group, community and social dimensions,
in addition to the individual user’s view (Internet of Everything).
Privacy and trust are essential for user’s to engage into the experiences that HCC
enables for them. HCC requires much more transparency on who or what is in
control or has certain rights. It remains to be seen whether current trends in big
data are going to provide a sufficient framework for this.
Another emerging feature of the HCC idea is that users are creative actors. In HCC
there is a sense that we put things together, more than that they are pre-packaged
technology bundles. The active involvement of the human as the composer or
bricoleur of its own technology-mediated experience is stronger, maybe even
more essential then the automated adaptation of the technology to the user. In
this sense HCC resonates well with current trends of ‘makers’, fablabs and social
innovation. Will HCC bring everyday creativity and popular cultural expression to
the heart of the research agenda, not just as a creative methodology to design new
technologies but as an integral feature of any human/socio-technologicalsystem?
Looking at this list, it is clear that HCC has not been achieved yet in a significant
way. The combination through technology of such features as personal empower-
ment, social enhancement, trust and user creativity has not yet been demonstrated in
a strong way. It is not even clear what form such a demonstration would take. Beyond
research, one can only speculate about the killer application that would demonstrate
its power. Maybe it could be in sports, in gaming or in fashion, where playfulness and
serious results can be combined and where wiling early adopters can be easily found.
What would be its first big market: commerce, entertainment, the grey economy, edu-
cation, the public sector or ecology? Or will it be and remain generic from the outset,
leaving it entirely to the users as creative actors to invent what it can do?
As the chapters of this book illustrate, Human Computer Confluence is a fascinat-
ing vision in which a lot remains to be understood. Some of the choices mentioned
above will need to be made more consciously in order to grow beyond the incidental
4Foreword
flashes of insight to a genuinely new paradigm of information technology. It has the
potential to create the opportunities for a distinct European approach to how humans
and technology can be combined.
References
EC. (2014). FP7: Human computer confluence (HC-CO). Retrieved 21 May 2015, from http://cordis.
europa.eu/fp7/ict/fet-proactive/hcco_en.html
HC2. (2013). Second HC2 Summer School, IRCAM, Paris. Retrieved 21 May 2015, from http://
hcsquared.eu/summer-school-2013
HLEG. (2004). Converging Technologies – Shaping the Future of European Societies. Brussels:
European Commission.
Section I: Conceptual Frameworks and Models
Alois Ferscha
1 A Research Agenda for Human Computer
Confluence
Abstract: HCI research over three decades has shaped a wide spanning research
area at the boundaries of computer science and behavioral science, with an impres-
sive outreach to how humankind is experiencing information and communication
technologies in literally every breath of an individual’s life. The explosive growth of
networks and communications, and at the same time radical miniaturization of ICT
electronics have reversed the principles of human computer interaction. Up until now
considered as the interaction concerns when humans approach ICT systems, more
recent observations see systems approaching humans at the same time. Humans and
ICT Systems apparently approach each other confluently.
This article identifies trends in research and technology that are indicative for the
emerging symbiosis of society and technology. Fertilized by two diametrically
opposed technology trends: (i) the miniaturization of information and communica-
tion electronics, and (ii) the exponential growth of global communication networks,
HCC over it’s more than two decades of evolution, the field has been undergoing three
generations of research challenges: The first generation aiming towards autonomic
systems and their adaptation was driven by the availability of technology to connect
literally everything to everything (Connectedness, early to late nineties). The second
generation inherited from the upcoming context recognition and knowledge process-
ing technologies (Awareness, early twentyhundreds), e.g. context-awareness, self-
awareness or resource-awareness. Finally, a third generation, building upon connect-
edness and awareness, attempts to exploit the (ontological) semantics of Pervasive
/ Ubiquitous Computing systems, services and interactions (i.e. giving meaning to
situations and actions, and “intelligence” to systems) (Smartness, from the mid twen-
tyhundreds). As of today we observe that modern ICT with explicit user input and
output are becoming to be replaced by a computing landscape sensing the physical
world via a huge variety of sensors, and controlling it via a plethora of actuators. The
nature and appearance of computing devices is changing to be hidden in the fabric
of everyday life, invisibly networked, and omnipresent, with applications greatly
being based on the notions of context and knowledge. Interaction with such globe
spanning, modern ICT systems will presumably be more implicit, at the periphery of
human attention, rather than explicit, i.e. at the focus of human attention.
Keywords: Humans and Computers, Pervasive and Ubiquitous Computing, Human
Computer Interaction, Ambient Intelligence, Socio-technical Systems
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License
© 2016 Alois Ferscha
8A Research Agenda for Human Computer Confluence
1.1 Introduction
Human Computer Confluence (HCC) has emerged out of European research initiatives
over the past few years, aiming at fundamental and strategic research studying how
the emerging symbiotic relation between humans and ICT (Information and Commu-
nication Technologies) can be based on radically new forms of sensing, perception,
interaction and understanding. HCC has been identified as an instrument to engage
an interdisciplinary field of research ranging from cognitive neuroscience, compu-
tational social sciences to computer science, particularly human computer interac-
tion, pervasive and ubiquitous computing, artificial intelligence and computational
perception.
The first definition of HCC resulted out of a research challenges identification
process in the Beyond-The-Horizon (FET FP6) effort:
“Human computer confluence refers to an invisible, implicit, embodied or even
implanted interaction between humans and system components. … should provide
the means to the user to interact and communicate with the surrounding environment
in a transparent “human and natural” way involving all sense…“
A working group of more than 50 distinguished European researchers structured
and consolidated the individual position statements into a strategic report. The key
issue identified in this report addressed fundamental research: “Human computer con-
fluence refers to an invisible, implicit, embodied or even implanted interaction between
humans and system components. New classes of user interfaces may evolve that make
use of several sensors and are able to adapt their physical properties to the current
situational context of users. In the near future visible displays will be available in all
sizes and will compete for the limited attention of users. Examples include body worn
displays, smart apparel, interactive rooms, large display walls, roads and architecture
annotated with digital information – or displays delivering information to the periph-
ery of the observers’ perception. Recent advances have also brought input and output
technology closer to the human, even connecting it directly with the human sensory and
neural system in terms of in-body interaction and intelligent prosthetics, such as ocular
video implants. Research in that area has to cover both technological and qualitative
aspects, such as user experience and usability” as well as societal and technological
implications “Researchers strive to broker a unified and seamless interactive frame-
work that dynamically melds interaction across a range of modalities and devices, from
interactive rooms and large display walls to near body interaction, wearable devices,
in-body implants and direct neural input and stimulation.
Based on the suggestions delivered with the Beyond-The-Horizon report, a consul-
tation meeting was called for “Human Computer Confluence” in November 2007, out
which resulted the FET strategy to “propose a program of research that seeks to employ
progress in human computer interaction to create new abilities for sensing, perception,
communication, interaction and understanding…“, and consequently the implementa-
tion of the respective research funding instruments like (i) new forms of interactive
Generations of Pervasive / Ubiquitous (P/U) ICT 9
media (Ubiquitous Display Surfaces, Interconnected Smart Objects, Wearable Com-
puting, Brain-Computer Interfaces), (ii) new forms of sensing and sensory perception
(New Sensory Channels, Cognitive and Perceptual Prosthetics), (iii) perception and
assimilation of massive scale data (Massive-Scale Implicit Data Collection, Navigating
in Massively Complex Information Spaces, Collaborative Sensing, Social Perception),
and (iv) Distributed Intelligence (Collective Human Decision Making, The Noosphere).
1.2 Generations of Pervasive / Ubiquitous (P/U) ICT
The novel research fields beyond Personal Computing could be seen as the early
‘seeds’ of HCC. Preliminarily suffering from a plethora of unspecific, competitive
terms like “Ubiquitous Computing” (Weiser et al., 1996), “Calm Computing” (Weiser
et al., 1996), “Universal Computing” (Weiser, 1991), “Invisible Computing” (Esler et
al., 1999), “Context Based Computing” (UCB, 1999), “Everyday Computing” (Abowd
& Mynatt, 2000), “Autonomic Computing”, (Horn, 2001), “Amorphous Computing”
(Servat & Drogoul, 2002), “Ambient Intelligence” (Remagnino & Foresti, 2005), “Sen-
tient Computing”, “Post-Personal Computing”, etc., the research communities con-
solidated and codified their scientific concerns in technical journals, conferences,
workshops and textbooks (e.g. the journals IEEE Pervasive, IEEE Internet Computing,
Personal and Ubiquitous Computing, Pervasive and Mobile Computing, Int. Journal
of Pervasive Computing and Communications, or the annual conferences PERVASIVE
(International Conference on Pervasive Computing), UBICOMP (International Confer-
ence on Ubiquitous Computing), MobiHoc (ACM International Symposium on Mobile
Ad Hoc Networking and Computing), PerComp (IEEE Conference on Pervasive Com-
puting and Communications), ICPCA (International Conference on Pervasive Com-
puting and Applications), ISWC (International Symposium on Wearable Computing),
ISWPC (International Symposium on Wireless Pervasive Computing), IWSAC (Inter-
national Workshop on Smart Appliances and Wearable Computing), MOBIQUITOUS
(Conference on Mobile and Ubiquitous Systems), UBICOMM (International Confer-
ence on Ubiquitous Computing, Systems, Services, and Technologies), WMCSA (IEEE
Workshop on Mobile Computing Systems and Applications), AmI (European Confer-
ence on Ambient Intelligence), etc. This process of consolidation is by far not settled
today, and more specialized research conferences are emerging, addressing focussed
research issues e.g. in Agent Technologies and Middleware (PerWare, ARM, PICom),
Privacy and Trust (STPSA, PSPT, TrustCom), Security (UCSS), Sensors (ISWPC,
Sensors, PerSeNS, Seacube), Activity Recognition and Machine Learning (e.g. IEEE
SMC), Health Care (PervasiveHealth, PSH, WiPH, IEEE IREHSS), Social Computing
(SocialCom), Entertainment and Gaming or Learning (PerEL).
Weiser’s seminal vision on the “Computer for the 21st Century” (Weiser, 1991) was
groundbreaking, and still represents the corner stone for what might be referred to
as a first generation of Pervasive/Ubiquitous Computing research, aiming towards
10A Research Agenda for Human Computer Confluence
embedded, hidden, invisible and autonomic, but networked information and com-
munication technology (ICT) systems (Pervasive / Ubiquitous ICT, P/U ICT for short).
This first generation definitely gained from the technological progress momentum
(miniaturization of electronics, gate packaging), and was driven by the upcoming
availability of technology to connect literally everything to everything (Connected-
ness, mid to late Nineties), like wireless communication standards and the exponen-
tially growing Internet. Networks of P/U ICT systems emerged, forming communica-
tion clouds of miniaturized, cheap, fast, powerful, wirelessly connected, “always on
systems, enabled by the massive availability of miniaturized computing, storage,
communication, and embedded systems technologies. Special purpose computing
and information appliances, ready to spontaneously communicate with one another,
sensor-actuator systems to invert the roles of interaction from human to machine
(implicit interaction), and organism like capabilities (self-configuration, self-healing,
self-optimizing, self-protecting) characterize this P/U ICT generation.
The second generation of P/U ICT inherited from the then upcoming sensor based
recognition systems, as well as knowledge representation and processing technologies
(Awareness, early Two Thousands), where research issues like e.g. context and situ-
ation awareness, self-awareness, future-awareness or resource-awareness reshaped
the understanding of pervasive computing. Autonomy and adaptation in this genera-
tion was reframed to be based on knowledge, extracted from low level sensor data
captured in a particular situation or over long periods of time. The respective “epoch
of research on “context aware” systems was stimulated by Schillit, Adams and Want
(Schilit et al., 1994), and fertilized by the PhD work of Dey (Dey, 2001), redefining the
term “context” as:
“…any information that can be used to characterize the situation of an entity.
An entity is a person, place, or object that is considered relevant to the interaction
between a user and an application, including the user and application themselves.”.
One result out of this course of research are autonomic systems (Kephart & Chess,
2003). Later, ‘autonomic elements’ able to capture context, to build up, represent and
carry knowledge, to self-describe, -manage, and -organize with respect to the environ-
ment, and to exhibit behavior grounded on knowledge based monitoring, analyzing,
planning and executing were proposed, shaping ecologies of P/U ICT systems, built
from collective autonomic elements interacting in spontaneous spatial/temporal con-
texts, based on proximity, priority, privileges, capabilities, interests, offerings, envi-
ronmental conditions, etc.
Finally, a third generation of P/U ICT was observed (mid of the past decade), build-
ing upon connectedness and awareness, and attempting to exploit the (ontological)
semantics of systems, services and interactions (i.e. giving meaning to situations and
actions). Such systems are often referred to as highly complex, orchestrated, coopera-
tive and coordinated “Ensembles of Digital Artifacts”. An essential aspect of such an
ensemble is its spontaneous configuration towards a complex system, i.e. a
Bibliography and Webliography 11
“... dynamic network of many agents (which may represent cells, species, indi-
viduals, nations) acting in parallel, constantly acting and reacting to what the other
agents are doing where the control tends to be highly dispersed and decentralized,
and if there is to be any coherent behavior in the system, it has to arise from competi-
tion and cooperation among the agents, so that the overall behavior of the system
is the result of a huge number of decisions made every moment by many individual
agents” (Castellani & Hafferty, 2009).
1.3 Beyond P/U ICT: Socio-Technical Fabric
Ensembles of digital artifacts as compounds of huge numbers of possibly heteroge-
neous entities constitute a future generation of socially interactive ICT to which we
refer to as Socio-Technical Fabric (late last decade until now), weaving social and
technological phenomena into the ‘fabric of technology-rich societies’. Indications of
evidence for such large scale, complex, technology rich societal settings are facts like
1012 -1013 “things” or “goods” being traded in (electronic) markets today, 109 personal
computer nodes and 109 mobile phones on the internet, 108 cars or 108 digital cameras
with sophisticated embedded electronics – even for internet access on the go, etc.
Todays megacities approach sizes of 107 citizens. Already today some 108 users are
registered on Facebook, 108 videos have been uploaded to YouTube, like 107 music
titles haven been labeled on last.fm, etc. Next generation research directions are thus
going away from single user, or small user group P/U ICT as addressed in previous
generations, and are heading more towards complex socio-technical systems, i.e.
large scale to very large scale deployments of ICT to large scale collectives of user up
to whole societies (Ferscha et al., 2012).
A yet underexplored impact of modern P/U ICT relates to services exploiting the
“social context” of individuals towards the provision of quality-of-life technologies
that aim for the wellbeing of individuals and the welfare of societies. The research
community is concerned with the intersection of social behavior and modern ICT,
creating or recreating social conventions and social contexts through the use of per-
vasive, omnipresent and participative technologies. An explosive growth of social
computing applications such as blogs, email, instant messaging, social network-
ing (Facebook, MySpace, Twitter, LinkedIn, etc.), wikis, and social bookmarking is
observed, profoundly impacting social behavior and life style of human beings while
at the same time pushing the boundaries of ICT simultaneously.
Research emerges aiming at understanding the principles of ICT enabled social
interaction, and interface technologies appear implementing “social awareness”, like
social network platforms and social smartphone apps. Human environment inter-
faces are emerging, potentially allowing individuals and groups to sense, explore and
understand their social context. Like the human biological senses (visual, auditory,
tactile, olfactory, gustatory) and their role in perception and recognition, the human
12A Research Agenda for Human Computer Confluence
“social sense” which helps people to perceive the “social” aspects of the environ-
ment, and allowing to sense, explore and understand the social context is more and
more becoming the subject of research.
Inspired by the capacity of human beings acting socially, in that shaping intel-
ligent societies, the idea of making the principles of social interaction also the
design, operational and behavioral principle of modern ICT recently has led to the
term “socio-inspired” ICT (Ferscha et al., 2012). From both theoretical and techno-
logical perspectives, socio-inspired ICT moves beyond social information processing,
towards emphasizing social intelligence. Among the challenges are issues of model-
ing and analyzing social behavior facilitated with modern ICT, the provision access
opportunities and participative technologies, the reality mining of societal change
induced by omnipresent ICT, the establishment of social norm and individual respect,
as well as e.g. the means of collective choice and society controlled welfare.
1.4 Human Computer Confluence (HCC)
Human Computer Confluence (HCC) has emerged out of European research initia-
tives over the past few years, aiming at fundamental and strategic research study-
ing how the emerging symbiotic relation between humans and ICT can be based on
radically new forms of sensing, perception, interaction and understanding (Ferscha,
2011). HCC has been identified as an instrument to engage an interdisciplinary field of
research ranging from cognitive neuroscience, computational social sciences to com-
puter science, particularly human computer interaction, pervasive and ubiquitous
computing, artificial intelligence and computational perception.
In order to further establish and promote the EU research priority on Human
Computer Confluence, research road-mapping initiatives were started (see e.g. HCC
Visions White Book, 2014), aiming to (i) identify and address the basic research prob-
lems and strategic research fields in HCC as seen by the scientific community, then
(ii) bring together the most important scientific leaders and industrial/commercial
stakeholders across disciplines and domains of applications to collect challenges
and priorities for a research agenda and roadmap, i.e. the compilation of white-books
identifying strengths, weaknesses, opportunities, synergies and complementarities
of thematic research in HCC, and ultimately to (iii) negotiate and agree upon strategic
joint basic research agendas together with their road-mapping, time sequencing and
priorities, and maintain the research agenda in a timely and progressive style.
One of these initiatives created the HCC research agenda book entitled “Human
Computer Confluence – The Next Generation Humans and Computers Research
Agenda” (HCC Visions White Book, 2014). It has been published under the acronym
“Th. Sc. Community” (“The Scientific Community”), standing for a representative
blend of the top leading scientists worldwide in this field. More than two dozens of
research challenge position statements solicited via a web-based solicitation portal
The HCC Research Agenda 13
are compiled into the book, which is publicly available to the whole scientific com-
munity for commenting, assessment and consensus finding. Some 200 researchers
(European and international) have been actively involved in this process. In addition,
the HCC Research Agenda and Roadmap is presented also in the format of a smart-
phone app, available in online (“The HCC Visions Book”-app). In the past 3 years
(2010–2013), the HCC community has become a 650 strong group of researchers and
companies working together to understand, not only the technological aspects of the
emerging symbiosis between society and ICT, but also the social, industrial, commer-
cial, and cultural impact of this confluence.
1.5 The HCC Research Agenda
The HCC research agenda as collected in “Human Computer Confluence – The Next
Generation Humans and Computers Research Agenda” (HCC Visions White Book,
2014) can be structured along the following trails of future research:
1.5.1 Large Scale Socio-Technical Systems
A significant trail of research appears to be needed along the boundaries where ICT
“meets” society, where technology and social systems interact. From the observation
how successful ICT (smartphones, mobile internet, autonomous driver assistance
systems, social networks, etc.) have radically transformed individual communica-
tion and social interaction, the scientific community claims for new foundations for
large-scale Human-ICT organisms (“superorganisms”) and their adaptive behaviors,
also including lessons form applied psychology, sociology, and social anthropology,
other than from systemic biology, ecology and complexity science. Self-organization
and adaptation as ways of harnessing the dynamics and complexity of modern,
networked, environment-aware ICT have become central research topics leading
to a number of concepts such as autonomic, organic or elastic computing. Existing
work on collective adaptive systems is another example considering features such
as self-similarity, complexity, emergence, self-organization, and recent advances in
the study of collective intelligence. Collective awareness is related to the notion of
resilience, which means the systemic capability to anticipate, absorb, adapt to, and/
or rapidly recover from a potentially disruptive event. Resilience has been subject to
a number of studies in complex networks and social-ecological systems. By creating
algorithms and computer systems that are modeled based on social principles, socio-
inspired will find better ways of tackling complexity, while experimenting with these
algorithms may generate new insights into social systems. In computer science and
related subjects, people have started to explore socially inspired systems, e.g. in P2P
networks, in robotics, in neuroscience, and in the area of agent systems. Despite this
14A Research Agenda for Human Computer Confluence
initial work, the overall field of socio-inspired system is still in an early stage of devel-
opment, and it will be one of future research goals to demonstrate the great potential
of social principles for operating large-scale complex ICT systems.
1.5.2 Ethics and Value Sensitive Design
ICT has become more sensitive to its environment: to users, to organizational and
social context, and to society at large. While ICT has largely been the outcome of a
technology-push focused on core computational functionality in the previous century
in the first place, it later extended to the users needs, usability, and even social and psy-
chological and organizational context of computer use. Nowadays we are approach-
ing ICT developments, where the needs of human users, ethics, systems of value and
moral norm, the values of citizens, and the big societal questions are in part driving
research and development. The idea of making social and moral values central to the
matrices, identity management tools) first originated in Computer Science at Stanford
in the seventies. This approach is now referred to as ‘Value Sensitive Design’ or ‘Values
in Design’. Among the most prevalent, ubiquitously recognized, and meanwhile also
socially pressing research agenda items relate to the concerns humans might have
in using and trusting ICT. Well beyond the privacy and trust related research we see
already today, the claim goes towards ethics and human value (privacy, respect,
dignity, trust) sensitivity already at the design stage of modern ICT (value sensitive
design), on how to integrate human value building processes into ICT, and how to
cope with ignorance, disrespectful, offending and violating humanvalues.
1.5.3 Augmenting Human Perception and Cognition
To escape the space and time boundaries of human perception (and cognition) via
ICT (sensors, actuators) has been, and continues to be among the major HCC research
challenges. Argued with the “Total Recall” prospect, the focused quests concern the
richness of human experience, which results not only from the pure sensory impres-
sions perceived via human receptors, but mostly from the process of identifying,
interpreting, correlating and attaching “meaning” to those sensory impressions. This
challenge is even posed to be prevalent over the next 50 years of HCC research. Dating
back to the “As we may think” idea (Bush & Think, 1945), claiming ICT to support,
augment or even replace intellectual work, new considerations for approaching the
issue are spawned by the technological, as well as cognitive modelling advance in
the context of brain computer interfaces. Understanding the operational principles
(chemo-physical), but much more than that the foundational principles of the human
brain at the convergence of ICT and biology poses a 21st century research challenge.
The ability to build revolutionary new ICT as a “brain-like” technology, and at the
The HCC Research Agenda 15
same time the confluence of ICT with the biological brain mobilizes great science not
only in Europe (see e.g. FTE flagship initiative HBP), but in neuroscience, medical and
computing research worldwide.
1.5.4 Empathy and Emotion
Humans are emotional, and at the same time empathic beings. Emotion and empathy
are not only expressed (and delivered via a subtle channel) when humans interact
with humans, but also when humans interact with machines. The ability to capture
and correlate emotional expressions by machines (ICT), as well as the ability of
machines (ICT) to express emotions and empathic behavior themselves is considered
a grand challenge of human computer confluence, while at the same time realism is
expressed on the potential success ever being possible towards it.
1.5.5 Experience and Sharing
With novel ICT, particularly advanced communication systems and ubiquitous net-
working, radically new styles of human-to-human communication -taking into
account body movements, expressions, physiological and brain signals- appear tech-
nologically feasible. With cognitive models about individuals, and digital representa-
tions of their engagements and experiences abstracted and aggregated from a com-
bination of stimuli (e.g. visual, olfactory, acoustic, tactile, and neuro-stimulation), a
new form of exchanging of these experiences and “thoughts” seem possible. Novel
communication experiences which are subtle, i.e. unobtrusive, multisensory, and
open for interpretation could build on expressions and indications of thoughts as
captured and related to a cognitive model on the sending side, encoded and trans-
mitted to a recipient through advanced communication media, and finally translated
into multimodal stimuli to be exposed to the receiving individual, and by that induce
mental and emotional states representing the “experience” of the sender.
1.5.6 Disappearing Interfaces
Seemingly in analogy to what was addressed at the turn of the century as the ICT
research challenge “The Disappearing Computer” (EU FP4, FP5), articulated as (i)
the physical disappearance of ICT, observed as the miniaturization of devices and
their integration in other everyday artefacts as, e.g., in appliances, tools, clothing,
etc. and (ii) mental disappearance, referring to the situation that artefacts with large
physical appearance may not be perceived as computers because people discern them
as (ICT also mentally move into the background) – appears to have come to a revival
16A Research Agenda for Human Computer Confluence
as far as notions of interfaces are concerned. As of today we observe that modern ICT
with explicit user input and output is becoming to be replaced by a computing land-
scape sensing the physical world via a huge variety of sensors, and controlling it via
a plethora of actuators. Since the nature and appearance of computing devices has
widely changed to be hidden in the fabric of everyday life, invisibly networked, and
omnipresent, “everything” has turned to become the interface: things and objects
of everyday use, the human body, the human brain, the environment, or even the
whole planet. Systems that happen to be perceived, understood and used explicitly
and intently as the interface tend to disappear. Implicit interaction replaces explicit
interaction.
1.6 Conclusion
Human computer confluence is a research area with the union of human brains and
computers at its center. Its main goal is to develop the science and technologies neces-
sary to ensure an effective, even transparent, bidirectional communication between
humans and computers, which will in turn deliver a huge set of applications: from
new senses, to new perceptive capabilities dealing with more abstract information
spaces to the social impact of such communication enabling technologies. Inevitably,
these technologies question the notion of interface between the human and the tech-
nological realm, and thus, also in a fundamental way, put into question the nature of
both. The long-term implications can be profound and need to be considered from an
ethical/societal point of view. Clearly, this is just a preliminary, yet evidenced classifi-
cation of HCC research from the solicitation process so far. As this research roadmap
aims to evolve continuously, some next steps of consolidation may possibly rephrase
and reshape this agenda, as new considerations, arguments and assessments emerge.
References
Abowd, G. D., & Mynatt, E. D. (2000). Charting past, present, and future research in ubiquitous
computing.ACM Transactions on Computer-Human Interaction (TOCHI),7(1), 29–58.
Bush, V., & Think, A. W. M. (1945). The atlantic monthly.As we may think,176(1), 101–108.
Castellani, B., & Hafferty, F. W. (2009). Sociology and complexity science: a new field of inquiry:
Springer Science & Business Media.
Dey, A. K. (2001). Understanding and using context. Personal and ubiquitous computing, 5(1), 4–7.
Esler, M., Hightower, J., Anderson, T., & Borriello, G. (1999, August). Next century challenges:
data-centric networking for invisible computing: the Portolano project at the University of
Washington. InProceedings of the 5th annual ACM/IEEE international conference on Mobile
computing and networking(pp. 256–262). ACM.
Ferscha, A. (2011). 20 Years Past Weiser: What’s Next?.IEEE Pervasive Computing, (1), 52–61.
Ferscha, A., Farrahi, K., van den Hoven, J., Hales, D., Nowak, A., Lukowicz, P., & Helbing, D. (2012).
Socio-inspired ICT.The European Physical Journal Special Topics,214(1), 401–434.
References 17
Ferscha, A., Lukowicz, P., & Pentland S. (2012). From Context Awareness to Social Awareness. IEEE
Pervasive Computing, 11(1), 32–41.
HCC Visions White Book. http://www.pervasive.jku.at/hccvisions/book/#home, last accessed July
18, 2014.
Horn, P. (2001). Autonomic computing: IBM\’s Perspective on the State of Information Technology.
Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer,36(1), 41–50.
Remagnino, P., & Foresti, G. L. (2005). Ambient intelligence: A new multidisciplinary
paradigm.Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions
on,35(1), 1–6.
Schilit, B., Adams, N., & Want, R. (1994, December). Context-aware computing applications.
InMobile Computing Systems and Applications, 1994. WMCSA 1994. First Workshop on(pp.
85–90). IEEE.
Servat, D., & Drogoul, A. (2002, July). Combining amorphous computing and reactive agent-based
systems: a paradigm for pervasive intelligence? In Proceedings of the first international joint
conference on Autonomous agents and multiagent systems: part 1(pp. 441–448). ACM.
Streitz, N., & Nixon, P. (2005). The disappearing computer.Communications of the ACM,48(3), 32–35
Weiser, M. (1991). The computer for the 21st century.Scientific american,265(3), 94–104.
Weiser, M., & Brown, J. S. (1996). Designing calm technology.PowerGrid Journal,1(1), 75–85.
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License
David Benyon and Oli Mival
2 Designing Blended Spaces for Collaboration
Abstract: In this paper, we reflect on our experiences of designing, developing, imple-
menting and using a real world, functional multi-touch enabled interactive collab-
orative environment (ICE). The paper provides some background theory on blended
spaces derived from work on blending theory, or conceptual integration. This is
applied to the ICE and results in a focus on how to deal with the conceptualization
that people have of new collaborative spaces such as the ICE. Five key themes have
emerged from an analysis of two years of observations of the ICE is use. These provide
a framework, TACIT, that focuses on Territoriality, Awareness, Control, Interaction
and Transitions in ICE type environments. The paper concludes by bringing together
the TACIT framework with the principles of blended spaces to highlight key areas for
design so that people can conceptualize the opportunities for creative collaboration
that the next generation of interactive blended spaces provide.
Keywords: Interaction Design, Collaboration, Multi-touch, Multi-surface Environ-
ment, Interactive Environments, Blended Spaces
2.1 Introduction
Blended spaces are spaces where a physical space is deliberately integrated in a
close-knit way with a digital space. Blended spaces go beyond mixed reality (Milgram
& Kishino, 1994) and conceptually are much closer to tangible interactions (Ishii &
Ullmer, 1997) where the physical and digital are completely coupled. The concept of
blending has been in existence for many years in the field of blended learning where
the aim is to design a learning experience for students that blends the benefits of
classroom learning with the benefits of distance, on-line learning, but more recently
the concept of blending has been applied to spaces and to interaction.
O’Hara, Kjelsko and Paay (O’hara, Kjeldskov, & Paay, 2011) refer to the distrib-
uted spaces linked by very high quality video-conferencing systems such as Halo as
blended spaces because of the apparently seamless joining of remote sites. In systems
such as Halo, great attention is paid to the design of the physical conference rooms
and to the angle and geometry of the video technologies in order to give the impres-
sion that two distant rooms are collocated. High-end video conferencing supports the
collaborative activity of business discussions, but it does not deal well with shared
information resources. O’Hara, et al. (O’hara et al., 2011) use the term blended interac-
tion spaces for ‘blended spaces in which the interactive groupware is incorporated in
ways spatially consistent with the physical geometries of the video-mediated set-up’.
Their paper explores the design of a blended interaction space that highlights the
© 2016 David Benyon, Oli Mival
Introduction 19
importance of the design of the physical space in terms of human spaces and the dif-
ferent proxemics (Hall, 1969) of personal spaces, intimate spaces and so on. They also
draw on a theory of spaces proposed by Kendal (1990) that describes the interactional
spaces that participants in collaborative activity share.
Jetter, Geyer, Schwarz and Reiterer (Jetter et al., 2012) also discuss blended inter-
action. They develop a framework for looking at the personal, social, workflow and
collaborative aspects of blended interaction. Benyon and Mival (Benyon & Mival 2013,
2012) describe the design of their interactive collaborative environment (ICE, dis-
cussed later in this chapter) focusing on the close integration of hardware, software
and room design to create new interactive spaces for creativity. A workshop at AVI 2012
conference on collaborative interactive spaces has led to a special issue of the journal
Personal and Ubiquitous Computing and another workshop at the CHI2013 confer-
ence on blended interaction. In these meetings researchers are developing theories
and examples of blended spaces and blended interactions in a variety of settings.
In addition to these examples of blended spaces and interaction in room environ-
ments, the idea of blended spaces has been applied to the domain of digital tourism
(Benyon, Mival and Ayan, 2012). Here the emphasis is on providing appropriate digital
content at appropriate physical places in order to provide a good user experience (UX)
for tourists. The concept of blending has also been used for the design of ambient
assisted living environments for older people (Hoshi, Öhberg, & Nyberg, 2011) and
for the design of products including a blood taking machine (Markussen, 2007) and a
table lamp (Wang, 2014).
A central feature about these latter examples is that they take as their theoreti-
cal basis the work of Fauconnier and Turner (Fauconnier, 1997; Fauconnier & Turner,
2002) on blending theory (BT), or conceptual integration. Originally developed as
a theory of linguistics and language understanding, BT has been applied to a huge
range of subject areas from mathematics to history to creativity (Turner, 2014) making
it more a general theory of creativity than just a theory of language construction. BT
is concerned with how people conceptualise new experiences and new ideas in terms
of their prior knowledge. Imaz and Benyon (Imaz & Benyon, 2007) originally applied
BT to Human-Computer Interaction (HCI) and Software Engineering (SE) looking at
how the metaphors in HCI and SE have changed over the years and how this effects
how these disciplines are perceived. In their book they analyse many examples of
user interfaces and interactive systems and suggest how BT could be used to design
interfaces and interactions.
The aim of this chapter is to develop the concept of blended spaces, in the context
of multi-user, multi-device, collaborative environments and to see the contribu-
tion that BT can make to the design of multi-device collaborative spaces. We see the
concept of a blended space as a clear example of human-computer confluence (HCC).
Benyon (Benyon, 2014) discusses how blended spaces provide a new medium that
enables new ways of doing things and of conceptualizing activities. He talks about
the new sense of presence that comes from blended spaces (Benyon, 2012). Our view
20Designing Blended Spaces for Collaboration
is that, in the days of blended spaces, designers need to design for interactions that
exploit the synergy between the digital and the physical space.
The particular space that is the focus of this work is known as the Interactive Col-
laborative Environment (ICE). The concept of the ICE took shape through an interdis-
ciplinary university project where the aim was to look at how the next generation of
technologies would change people’s ways of working. We envisaged a ‘future meeting
room’ that would contain the latest multi-touch screens and surfaces, novel software
and work with mobile and tangible technologies to create a new environment for
meetings of all types including management meetings, research meetings, collabora-
tive compositing for magazines, video-conferencing and creative brainstorming meet-
ings. We negotiated a physical space for the meeting room and within a limited budget
commissioned a multi-touch boardroom table and installed five multi-touch screens
on three of the walls to produce a meeting room with an interactive boardroom table
that can seat 10 people, interactive whiteboard walls and five wall mounted multi-
touch screens (Figure 2.1 ).
During the last two years the ICE has been used for meetings of all kinds by
people from all manner of disciplinary backgrounds. This activity has been observed
and analyzed in a variety of different ways, resulting in a set of five generic themes
that designers of such environments need to consider. These provide our framework
for describing the design issues in ICE environments, TACIT, that stands for territorial-
ity, awareness, control, interaction and transitions.
The chapter is organized as follows. Section 2 provides a background to the work
on conceptual integration, or blending theory. Section 3 describes the design ratio-
nale for the ICE in the context of the design of the physical space and the design of the
digital space. We discuss how physical and digital spaces are blended to create a new
space with its own emergent properties. It is within this blended space that people
must develop their conceptual understanding about the opportunities afforded by
the blended space and how this can alter their ways of working. Section 4 provides
an analysis of the ICE in use, drawing upon observations over a period of two years,
a controlled study of the room being used and interviews with users of the ICE. This
analysis highlights five key characteristics of cooperative spaces such as the ICE that
reflect our own experiences and many of the issues raised elsewhere in the literature.
In section 5 we see how these characteristics are reflected in the physical, digital and
conceptual spaces that make up the blended space. Section 6 provides some conclu-
sions for designing blended spaces such as the ICE.
Blending Theory 21
Figure 2.1: The Interactive Collaborative Environment (ICE)
2.2 Blending Theory
Fauconnier and Turner’s book The Way We Think (Fauconnier & Turner, 2002) intro-
duced their ideas on a creative process that they called conceptual integration or
blending. They argued that cognition could be seen in terms of mental spaces, or
domains. Cognition involves the projection of concepts from domains and their inte-
gration into new domains. Blending Theory (BT) develops and extends the ideas of
Lakoff and Johnson on the importance of metaphor to thought (Lakoff & Johnson,
1980), (Lakoff & Johnson, 1999). Where metaphor is seen as a mapping between
two domains, Fauconnier and Turner see blending in terms of four domains. They
explain the simple but powerful process as follows (see Figure 2.2). Two input spaces
(or domains) have something in common with a more generic space. Blending is an
operation that is applied to these two input mental spaces which results in a new,
blended space. The blend receives a partial structure from both input spaces but has
an emergent structure of its own.
In linguistics blending theory is used to understand various constructs such as
counterfactual statements, metaphors and how different concepts arise (Fauconnier,
1997). For example blending theory can be used to understand the difference between
houseboat and boathouse by looking at the different ways in which the concepts of
house and boat can be combined. There is now extensive work on blending theory
applied to all manner of subjects that offer different insights into the way we think.
Mark Turner’s web site is a good starting place (Turner, 2014).
The main principles of blending are that the projections from the input spaces
create new relationships in the blend that did not exist in the original inputs, and that
our background knowledge in the form of cognitive and cultural models allow the
composite structure to be experienced in a new way. The blend has its own emergent
logic and this can be elaborated to produce new ideas and insights. This blended
space may then go on to be blended with other mental spaces.
22Designing Blended Spaces for Collaboration
Figure 2.2: Concept of Blend
Fauconnier and Turner (Fauconnier & Turner, 2002) discuss different types of blend
and provide guidance on what makes a good blend. Four types of blend are identi-
fied based on the way in which concepts from the input spaces are projected into the
blended space, from simple one-to-one mappings to more complex ‘double scope’
blends that creatively merge concepts from the input domains to produce a new expe-
rience. Fauconnier and Turner see the process of blending as consisting of three main
processes. Composition is the process of understanding the structure of the domains,
completion is the process of bringing relevant background knowledge to the process
and elaboration is the process of making inferences and gaining insight based on
these relationships. They propose six guiding principles to support the development
of blends. The first three, integration, web and unpacking, concern getting a blend
that is coherent and in a form that people can understand where the blend has come
from. The fourth, topology, concerns the correspondences between the input spaces.
The fifth, ‘good reason’, captures the generic design principle that things should only
be in the blend if there is a good reason for them to be there and the sixth, metonymic
tightening, concerns achieving a blend that does not have superfluous items in it.
Imaz and Benyon (2007) have applied the ideas of conceptual blending to analyze
developments in HCI and software engineering. They analyze a number of well-
known examples of user interfaces, including the trash can icon and the device eject
function in Mac OS and critical HCI concepts such as scenarios and use cases. One
example they consider is the concept of a window in a computer operating system.
This is a blend of one mental space – the concept of a window in a house – and
another mental space, the concept of collecting some computer operations together.
The generic space is the idea of being able to see a part of a large object; a window on
the world if you like. The process of bringing these spaces together results in a new
concept of ‘window’ that now includes things such as a scroll bar, a minimize button
Blending Theory 23
and so on that you would not associate with a window in a house. The blended space
of a computer window has inherited some shared characteristics from the generic
space of ‘looking onto something’, but now has its own characteristics and emergent
properties. This can be illustrated as in Figure 2.3.
Imaz and Benyon argue that in interaction design, designers need to reflect and
think hard about the concepts that they are using and how these concepts affect their
designs. They emphasize the physical grounding of thought by arguing that design-
ers need to find solutions to problems that are ‘at a human scale’. Drawing upon
the principles of blends suggested by Fauconnier and Turner they present a number
of HCI design principles. When creating a new interface object, or new interactive
product designers will often create a blend from existing designs. Designers should
aim to preserve an appropriate topology for the blend, allowing people to unpack the
blend so that they can understand where the new conceptual space has come from.
There are principles for compressing the input spaces into the blend, aiming for a
complete structure that can be understood as a whole (the integration principle) and
for keeping the blend relevant and at a human scale.
They go on to present an abstract design method that shows how conceptual
blending can be used during the analysis phase of systems development to under-
stand the issues of a particular situation and how it can be used during the design
stage to produce and critique design solutions. For example, they discuss the exis-
tence of the trashcan on the Windows desktop. Here the designers have chosen not to
enforce the topology principle (which would suggest in this case that since trash cans
normally go underneath a desk the trash can should not be on the desk top). Instead
the designers have promoted the integration principle, keeping the main functions of
the interface together in a ‘desk top’ metaphor.
Figure 2.3: Blend for a computer window
24Designing Blended Spaces for Collaboration
The danger in presenting blending theory in such a short section is that it can seem
trivial, when in fact it is a very powerful idea. Underlying BT are ideas about embod-
ied cognition that go back to the roots of how human cognition starts with the bodily
experiences that we have as human babies growing up in a three dimensional world.
Lakoff and Johnson (1999) develop this ‘philosophy of the flesh’ from a conceptual
perspective and Rohrer (2010) develops related ideas from a cognitive science and
neurological perspective. People think the way they do because of their inherent
physical and perceptual schemata that are blended to produce new concepts that in
their turn are blended to form new concepts. Understanding this background helps
designers to create experiences ‘at a human scale’.
2.3 Blended Spaces
In looking at the sense of presence in mixed reality spaces, Benyon (Benyon, 2012)
develops a view of digital and physical spaces in terms of four characteristics; ontol-
ogy, topology, volatility and agency. These constitute the generic space structure
that both physical and digital spaces share. Bringing blending theory together with
the idea of physical and digital spaces leads to the position illustrated in Figure 2.4.
He argues that for the purpose of designing mixed reality experiences, physical and
digital space can be usefully conceptualized in terms of these four key characteristics.
The ontology of the spaces concerns the objects in the spaces. The topology of
the spaces concerns how those objects are related to one another. The dynamics or
volatility of the spaces concerns how elements in the spaces change over time. The
agency in the spaces concerns the people in the spaces, the artificial agents and the
opportunities for action in the spaces. By understanding these characteristics and
looking at the correspondences between the physical and the digital spaces, design-
ers will produce new blended spaces that have emergent properties. In these spaces,
people will not be in a physical space with some digital content bolted on. People will
be present in a blended space and this will give rise to new experiences and new ways
of engaging with the world.
The conceptualization of blended spaces illustrated in Figure 2.4 relies on a
generic way of talking about spaces – ontology, topology, volatility and agency. This
is the generic space of spaces and places that is projected onto both the physical
and the digital spaces. The correspondences between the physical and the digital
are exploited in the design of the blended space. The job of the designer is to bring
the spaces together in a natural, intuitive way to create a good user experience. The
designer should design the blended space according to the principles of designing
with blends such as drawing out the correspondences between the topology of the
physical and digital spaces, using the integration principle to deliver a whole experi-
ence and designing at a human scale.
Blended Spaces 25
Figure 2.4: Conceptual blending in mixed reality spaces
Another consideration that is important in the design of blended spaces is that the
physical and the digital spaces rarely co-exist. There are anchors, or touch points,
where the physical is linked to the digital, but there are many places where the physi-
cal and the digital remain separate. QR codes or GPS are examples of anchor tech-
nologies that bring the physical and the digital together. In the context of an ICE type
environment people will be collaborating through some physical activity such as
talking to each other, but then may access the digital space to bring up some media
to illustrate what they are saying. The conversation may then continue with shared
access to the digital content. In the context of digital tourism we may observe someone
walking through a physical space, accessing some digital content on an iPad and
then continuing his or her physical movement. Thus, in blended spaces, people move
between the physical, the digital and the blended spaces. This movement through
and between spaces is an important part of the blended space concept and leads on
to issues of navigation in physical, digital and blended spaces. It appears in Benford’s
work on spectator interfaces as the idea of a hybrid trajectory (Benford, Giannachi,
Koleva, & Rodden, 2009). Related ideas from Yvonne Rogers on access points, or entry
points are discussed later.
The blended space encompasses a conceptual space of understanding and making
meaning and this is where the principles of designing with blends are so important.
People need to be aware of both the physical and the digital spaces, what they contain
and how they are linked together. People need to understand the opportunities
afforded by the blended space and to be able to unpack the blend to see how and why
the spaces are blended in a particular way. People need to be aware of the structure of
the physical and the digital, so that there is a harmony; the correspondences between
26Designing Blended Spaces for Collaboration
the objects in the spaces. The overall aim of blended spaces is to design for a great
UX, at a human scale.
2.4 Designing the ICE
The aim of the project to develop the ICE was to provide a great experience for people
and to give them an insight on what leading edge meeting spaces can be like. The aim
is to generate enthusiasm and to get people to see how new spaces could be used by
domain experts in very different areas and the impact that these spaces will have on
their working practices.
As with all real-world projects, the ICE had to comply with a number of con-
straints such as the existence of a room and a budget. It was also to be a ‘bring your
own device’ (BYOD) environment. The philosophy underlying the design focused on
providing an environment that would help people within it fulfill their activities and
do so in pleasurable intuitive ways. Wherever possible the aim is to remove function
from the content of devices (screens, laptops, mobile devices) and instead consider
these devices as portals onto function and content that is resident in a shared space.
This should enable and facilitate real time, concurrent, remote collaboration. Another
key aim is to enable the seamless mixing of digital and analogue media. People bring
notebooks, pens and paper to meetings and we are keen that such analogue media
should co-exist happily alongside the digital spaces.
2.4.1 The physical Space
The physical space that was available for the ICE was an empty office, so the design
started with a room, a vision and a budget of €150,000 about a third of which went on
technologies for the digital space, a third on room alterations and a third on necessary
infrastructure. After extensive research into the options available we settled on the
following technologies. A 46” n-point HD (1080p) multi touch LCD screen mounted
on the end wall of the room. This screen uses the diffused illumination (DI) method
for detecting multi-touch and is capable of detecting finger and hand orientation as
well as distinguishing between different users’ hands. A 108” n-point multi-touch rear
projection boardroom table, also using DI is the centrepiece of the room. The table
can recognize and interact with objects placed on its surface such as mobile phones,
laptops or books using infrared fiducial markers. It has 2 patch panels on either side
allowing the connection of USB peripherals and storage devices as well as any exter-
nal DVI or VGA video source, such as a laptop or tablet, to any of the surfaces.
The table was designed specifically for the space available, which determined
its dimensions and technical specification. Due to the requirement of using the table
both when sitting and standing the surface is 900mm from the floor, the standard
Designing the ICE 27
ergonomic height for worktops. Four 42” HD (1080p) dual point multi-touch LCD
screens utilizing infrared overlays are mounted on the room’s side walls. Each screen
is driven by a dedicated Apple Mac Pro, which can triple boot into Mac OSX, Windows
7 and Linux. The room has 8-channel wireless audio recording for 8 wearable micro-
phones as well as 2-channel recording for 2 room microphones. Each audio channel
can be sent individually to any chosen destination (either a computer in the ICE or
to an external storage via IP) or combined into a single audio stream. A webcam
allows for high definition video (1080p) to be recorded locally or streamed over IP.
The recording facility is used both as a means of archiving meetings (for example,
each audio channel can record an individual participant) or for tele- and video confer-
encing activities. Any recordings made can be stored both locally and on an external
cloud storage facility for remote access.
The walls are augmented with the Mimio™ system to serve as digital white-
boards, thus when something is written on the whiteboard it is automatically digitally
captured. Importantly no application needs to be launched before capture begins, the
process of contact between the pen and the wall initiates the digital capture.
2.4.2 The Digital Space
The ICE hardware has been selected to offer the widest range of development and
design opportunities and as such we have remained platform agnostic. To achieve
this, all the hardware runs on multiple operating systems (Mac OS X, Windows 7 & 8,
Linux). Alongside the standard development environments are the emerging options
for multi-touch, central to which is the Tangible User Interface Objects (TUIO) proto-
col, anopen framework that defines a common protocol and API for tangible multi-
touch surfaces. TUIO enables the transmission of an abstract description of interac-
tive surfaces, touch events and tangible object states from a tracker application and
sends it to a client application.All the hardware in the room is TUIO compliant. The
TUIO streams generated by the screens and table in the ICE are passed to the com-
puters via IP thus enabling the possibility for more extensive remote screen sharing.
Indeed, if a remote participant has the capacity to generate a TUIO stream of their
own, as is the capability with many modern laptop track-pads, they can collaborate
with the ICE surfaces both through traditional point and click methods but also multi-
touch gestures.
A key design issue for the software is that there is no complex software archi-
tecture required. All applications are available at the top level; the TUIO protocol
means that any device running it can interact with other devices running it. This
allows for easy integration of mobile devices into the room and sharing control across
devices. Since everything in the room is interconnected through the Internet, the
room demonstrates how such sharing of content and manipulation of content could
be physicallydistributed.
28Designing Blended Spaces for Collaboration
In terms of software, the ICE makes extensive use of freely available applications
such as Evernote™, Skype™, etc. A key design decision for the ICE is to keep it simple
and leverage the power of robust 3rd party services. People need to be able to come
into the room and understand it and to conceptualize the opportunities it offers for
new ways of working. It is little use having applications buried away in a hierarchical
menu system where they will not be found. Hence the design approach is to make the
applications all available at the top level. One particularly important application is
Dropbox™, which is used to enable the sharing of content across devices. Since the
Dropbox™ service is available across most devices; any content put into one device is
almost immediately available on another. For example, take a photo on an iPhone or
Android Phone and it is immediately available on the table and wall screens. Essen-
tially Dropbox™ works as the synching bridge across all the separate computers
driving each surface, which enables a seamless “shared repository” experience for
users. Any file they utilize on one surface is also available on any other, as well as
remotely should the user have the appropriate access authority.
Of course things are constantly changing, but the aim is to keep this
philosophystable.
2.4.3 The Conceptual Space
The conceptual space concerns how people understand what the novel blend of phys-
ical and digital spaces allows them to do and how they frame their activities in the
context of that understanding. There are a lot of novel concepts to be understood. For
example, people need to understand that the screens on the walls are not comput-
ers, but are windows onto data content in the cloud. They will recognise the Inter-
net Explorer icon on a screen and rightly conceptualise that this gives them Internet
access, but they may not realise that they can save files to a shared Dropbox™ folder
and hence enable files to be viewed through different screens. They need to concep-
tualise the wall screens as input and output zones that can show different views at
different times. People who have used the ICE a few times come to understand that
they can see an overview of some data set on one screen and the detail on another and
that this is useful for particular activities. The wall screens can be easily configured to
mirror each other, or to show separate content. The challenge for spaces such as the
ICE is that people do not have a mental model, a conceptualization, of the space when
they first arrive. It is up to the designers to present an informative, coherent image of
the system that enables people to develop an appropriate conceptual understanding
of the opportunities. For example we developed a control room map to help people
conceptualise the interaction between the different screens and how any screen
could be the source of an image displayed on any number of other screens. This is
one example of an attempt to provide a way into the conceptual space. Restricting
The TACIT Framework 29
the functionality of the digital space and providing this through a few familiar icons
is another.
2.4.4 The Blended Space
We can summarise the ICE through the lens of conceptual blending as illustrated in
Figure 2.5.
2.5 The TACIT Framework
We have now had the opportunity to observe many meetings in the ICE. It is a space
that encourages people to get up and move to one of the screens to demonstrate some-
thing, or use the whiteboard capabilities of the wall surface to illustrate ideas. The
room is highly flexible in terms of lighting, audio and the use of the blinds. Thus the
atmosphere can be quickly changed as required. The ability of people to easily join in
the room using internet-based video-conferencing has proved an important feature.
The room certainly encourages collaboration and participation. With five screens
it is very natural to have documents displayed on one or two, web searches on another
and perhaps YouTube videos on another. However, it is also true that many people
find it difficult to conceptualize what the room can do and how they can make it do
things. This serious challenge for interaction design – how to get people to under-
stand the opportunities afforded by novel interactive spaces – remains stubbornly
difficult to solve.
Figure 2.5: The ICE as a blended space
30Designing Blended Spaces for Collaboration
The meetings that we have observed have included conference organisation meet-
ings, preparing research grant proposals, publicity preparation, teaching interaction
design, remote viva voce examinations and numerous general business meetings. A
controlled study of the ICE has been completed that included a grounded approach
to analyzing video, interview and questionnaire data and a survey of user attitudes
to the ICE has been completed. The results of this analysis of video, the interviews
and the real-time observation have provided a rich picture of collaboration in the ICE.
Classic issues of computer supported cooperative work have been identified and
described over the years (Ackerman, 2000). Against this background five themes have
emerged from our analysis of the ICE in use that provide a way of structuring the
issues that have arisen and that designers of room environments need to deal with.
These reflect the main concerns identified in previous literature on collaborative envi-
ronments. We discuss the issues in terms of territoriality, awareness, control, interac-
tion and transitions between spaces: the TACIT framework.
2.5.1 Territoriality
Territoriality concerns spaces and the relationships between people and spaces. Scott
and Carpendale (Scott & Carpendale, 2010) see territoriality as a key issue in the use
of tabletops (both traditional and interactive) for cooperative work. They identify per-
sonal spaces, group spaces and storage spaces. They also point to the importance of
orientation, positioning and proximity in addition to the partitioning of spaces into
different regions. These all contribute to people’s sense of territory and their relation-
ships with their space. Territory is also important on large multiuser displays (Pel-
tonen et al., 2008). There are many social and psychological characteristics of ter-
ritories that designers need to consider such as proxemics (Hall, 1969), (Greenberg,
Marquardt, Ballendat, Diaz-Marino, & Wang, 2011) and how comfortable people feel
when they are physically close to others.
Issues of territoriality are central to working in the ICE and we have witnessed
many examples of groups configuring and reconfiguring their spatial relations
depending on the task. We have observed incidents where people cluster around
a wall screen watching one person working and commenting on what is going on
before going back to work individually. People have commented that being able to
work in different places in the room helps collaboration and moving around in the
room makes collaboration easier.
We have seen a number of incidents where the assignment of roles and tasks is
influenced by location and proximity to a particular piece of technology. The role
of penholder in brainstorming tasks may be assigned to the person sitting nearest
to the whiteboard. Participants tend to use the wall-screen nearest to them and the
person interacting with the tabletop applications at the table was the person sitting
nearest the controls at the bottom edge of the screen. There is often a close connection
The TACIT Framework 31
between the control of physical space in the environment and the control of screen
workspace, which in turn affects the assignment of roles and tasks.
The tabletop can be configured in different ways and there is one setting that has
six individual places each with a digital keyboard and media browser. This allows
people to have their personal space and then to move work into a public sphere
when they are ready. Haller, et al., (2010) emphasise the importance of these differ-
ent spaces in their NiCE environment. Personal spaces are provided through individ-
ual PCs and through personal sketching spaces. These can be shared with the group
through a large wall display. In earlier contributions to collaborative spaces, Buxton
(2009) describes media space in terms of the person space, task space and reference
space and provides a number of design guidelines for ensuring quality spaces. It is
the intersection of these and the ease of moving between them that is important.
2.5.2 Awareness
The issue of awareness is a common central theme running through the literature on
cooperative work (e.g. (Tang, 1991)) and is a central issue for the ICE. The concept of
awareness hides a large amount of complexity. Schmidt (Schmidt, 2002) lists seven
common adjectives often attached to the word ‘awareness’ such as peripheral, back-
ground, passive, reciprocal and so on. He goes on to explore the concept in detail,
finally arguing that “the term ‘awareness’ is being used to denote those practices
through which actors tacitly and seamlessly align and integrate their distributed and
yet interdependent activities”. Awareness includes aspects of attention and focus, so
explicit awareness of what others have done is also an important aspect. Awareness
is much more than simple information, however. It is to do with the social aspects of
how people align and integrate their activities. Awareness is an attribute of an activity
that relates to many of the characteristics of a situation. Rogers and Rodden (Rogers
& Rodden, 2003) also emphasise the importance of dealing with different types of
awareness and with the transitions between them (see below).
In the ICE, different tasks invite different degrees of awareness. For example,
brainstorming tasks require shared attention whereas an individual searching the
web on a wall screen does not. Tasks that can be undertaken in parallel may need
support to keep others aware of progress and when the task is completed. Several
people have commented that a shared Dropbox™ folder is useful for maintaining
awareness as it gives real time information on the progress of the work of the other
participants; images dragged into Dropbox™ on a screen soon appear in Dropbox™
on the table.
Even during individual tasks there are frequent shifts between shared and divided
attention with people participating in discussions in a group across the room and
then turning their backs to individually search for images and information on the
wall screens.
32Designing Blended Spaces for Collaboration
When collaborating in a shared physical environment like the ICE participants
become aware of each other’s activity through direct perception. People are able to
see and hear each other as they work at the surfaces, moving around the room or
talking to other participants. Writing on the whiteboards enhances awareness and
collaboration and allows people to use gestures to refer to specific objects or the
organisation of objects on the surfaces.
The various surfaces support collaboration through awareness differently. There
have been instances where participants grew silent while working on the wall screens
and focused on their own tasks loosing awareness of what was going on in the room
behind them. Working around the table creates a higher degree of shared awareness;
one of the advantages of table displays with respect to collaboration.
2.5.3 Control
Control refers both to the control of the software systems and to social control of the
collaborative activity. Yuill and Rogers (Yuill & Rogers, 2012) highlight the importance
of control in their discussion of collaborative systems. This is related to the concepts
of access points (see below). Rogers and Lindley (Rogers & Lindley, 2004) also iden-
tify control as one of their factors describing collaborative environments along with
awareness, interaction, resources, experience, progression, team and coordination.
In the ICE we have observed people taking turns to interact with the table with
only one person touching it at a time in order to maintain control. However, some
of the software on the table has a lack of support for multi user interaction and this
creates issues with conflict of control. It is easy for one person to enlarge a display
(often accidentally) so that it covers the whole of the tabletop and the whole work-
space and all the objects on it are covered over. People have adapted to this by taking
turns interacting with the table.
A common configuration of the input and output structure is to have two screens
mirroring each other on opposite sides of the table. However, whilst this helps aware-
ness, it can negatively impact control. One person may have started interacting with
a screen without realising that another person was using the mirrored screen and
hence takes over control of the pair of mirrored screens.
Often people sitting at the “bottom side” of the interface near the controls were
also the ones who controlled and interacted with the application. However we also
observed incidents where people are working at the table from both sides without
regard to the orientation of the application. The location of the controls on the
bottom edge at the lower left and right side of the screen demonstrated a “short arm
problem”. Due to the size of the table’s screen, it is impossible for someone standing
at the centre of an edge of the screen to reach the corners without actually moving his
or her body to the left or right. This also gives rise to a mode of collaboration where the
participant standing near to the controls would press buttons at appropriate times.
The TACIT Framework 33
Control is a key component of Yuill and Rogers framework for collaboration. The
locus of control is closely related to the access points provided by the technology.
Their main focus is on aiming for equitable control in a collaborative setting and they
emphasise that the location of access points and how people move between access
points is critical.
Control, and people’s understanding of what they can control is key to ideas of
appropriation and ‘tailoring culture’. People need to be encouraged to understand that
tailoring is possible in an ICE organizationally, technically and socially. This tailoring,
or customization is an essential part of the appropriation of technology and of spaces.
People need to adapt, and shape the environment, the interactions, the content and
the computational functions that are available to suit their ways of working. However,
this is easier said than done. People have to be confident and capable to appropriate
their technological spaces. They need a conceptual understanding of the digital and
physical spaces so that they can change their ways of working, and designers need to
design to support appropriation and the formation of an appropriate conceptualiza-
tion of the space.
2.5.4 Interaction
This theme concerns how people interact with each other and how they interact with
the activities they are undertaking. In collaborative tasks, articulation is concerned
with how work is divided up and tasks are allocated between members of a group.
How this happens is quite different with respect to how the surfaces of the room are
used for the different tasks. The users’ familiarity with the room, their experience
with working on touch screens, their prior knowledge of each other and experience
in working together are all factors in producing the different conceptualizations of
the ICE and hence the best ways to interact. People do things in different ways and
the variation in the use of the room and its facilities indicates that the environment
effectively supports this variety.
The ICE supports articulation as it affords the distribution and execution of sub-
tasks. The physical layout of the room gives easy access to workspace and this makes
coordinating tasks easy. We have also seen numerous and at times very rapid shifts
in the social organisation of work. For example, people will shift between working
individually at the wall screens and turning around to communicate across the room
as a group. Shifts also occur when people go to the wall screens to search web for
information during a discussion or the execution of a task only to return to the group
when information is found. Also while working at the table there is a variety of ways
to organise the work. The parallel tasks of searching and sorting can be accelerated if
the group divides itself into subgroups. This may be contrasted with serial tasks such
as putting collaborative projects together.
34Designing Blended Spaces for Collaboration
We have observed tasks being distributed by negotiation through verbal commu-
nication, but also seen people spontaneously go to a surface and start doing a task.
We also observed how roles and tasks were handed over simply by turning around
workspace on the table surface.
Interaction is one of the four components in the model of collaboration discussed
by O’Hara et al. (O’Hara et al., 2011) along with work, communication and service.
They emphasise the importance of getting the granularity of the space right. The
interplay between the spatial configuration, the social organisation and interaction is
critical. O’Hara, et al. discuss their blended interaction space (BISi) as an example of
workplace design, once again emphasising proxemics in design and the control and
set up of work tools as an essential part of the interaction
Hurtienne and Israel (2013) offer sound design advice for designing ICE type envi-
ronments with their PIBA-DIBA technique. PIBA-DIBA stands for ‘physical is best at
– digital is best at’. The aim is to identify where designers should allocate parts of the
interaction to physical objects and where they should be allocated to digital objects.
For example physical objects are perceived in 3D and are graspable whereas digital
objects are easy to transfer and transform into other representations. The aim of PIBA-
DIBA is to sensitize designers to the qualities of the different media in order to design
for better interaction in blended spaces.
2.5.5 Transitions
Blended spaces are rarely completely integrated. Instead there are touch points where
the digital and physical are brought together and where people transition from the
digital to the physical or vice versa. In the context of digital tourism, for example, the
global positioning system (GPS) is often used to link the physical world of a tourist
location with some digital content that is relevant to that location. Quick Response
(QR) codes also serve this purpose. In the context of blended spaces such as the ICE
people transit from the physical to the digital and back again as they use personal
and shared devices to access digital content and bring this into the physical world of
displays and discussion.
Discussions with users of the ICE indicate that the physical layout of the room
gives easy access to different workspaces. It seems to be less clear whether the indi-
vidual platforms (whiteboard, wall screens or table) were easily accessible for all
people. For example, the placement of the whiteboards at the corners of the room
has been suggested as a problem limiting the physical access to the board, and others
have argued that seeing objects from different perspectives when working at the table
is a problem.
Transitions between spaces and between the physical and the digital are identi-
fied by Scott, Grant and Mandryk (Scott, Grant, & Mandryk, 2003) as a significant
feature of tabletop interaction. Yvonne Rogers and her colleagues in a number of
Discussion 35
publications describe these transitions in terms of access points, or entry points. They
find that people are excluded from equitable participation in collaborative activities
because of difficulties in gaining access to the physical environment, or the digital
environment or both, or in moving between the physical and the digital. In the Shared
Information Space framework they offer advice on removing barriers to access and
enabling entry points that provide a good overview of the space and the opportunities
to move between locations in the spaces.
2.6 Discussion
People are aware of each other’s actions because they can see and hear each other
and they can sense what is going on in the room. People are able to move about and
interact with physical and virtual objects and information in the room and on the
surfaces. They are aware of each other as human beings and react to each other and
the physical environment according to norms and habits they find appropriate for the
situation. The physical and social interactions are intricately intertwined.
Visibility of information and objects in the environment — often referred to as the
information and interaction resources — is another key finding. Different surfaces
offer different opportunities for reference. People check notes on the whiteboards or
use file browser windows to follow the progress of work or to recapture information
needed for the work being done. These notes and traces of work were important tools
in the collaborative work as they serve as external memory and a representation of
common ground (Monk, 2003).
The shifts in social organisation of work are important. We have observed con-
stant fluctuations of groups of participants being formed and broken up to work on
tasks, communicate or just to observe the progress of a task. There are shifts between
working collaboratively in groups and working alone on subtasks. The fluency in the
social organisation of work reflects the way participants move around in the physical
environment and take control of physical and interactional resources.
2.6.1 Designing the Physical Space
The design of physical space draws upon the disciplines of architecture and interior
design and an understanding of social psychology. The physical space comes with
the social expectations of people concerning behaviours and at the same time the
physical space will help to shape those behaviours. The physical space allows for
interaction between people through touch, proximity, hearing and visual perception.
The physical space influences many aspects of the interaction such as territory, posi-
tion, control, and orientation. Orientation is an issue on flat surfaces and occlusion
needs to be avoided.
36Designing Blended Spaces for Collaboration
2.6.2 Designing the Digital Space
The digital space concerns data and how it is structured and stored. It concerns the
content that is available and the software that is available to manipulate the content.
The digital space is the medium through which people engage with content. Digital
tools need to be appropriate for the physical space and for the characteristics of the
devices that interface with the digital space. Gestures, touch and other forms of phys-
ical interaction provide a direct engagement between the physical and the digital.
With no standards for gestures on multi-touch surfaces, designers need to attend to
facilitating learning and minimizing mistakes.
2.6.3 Designing the Conceptual Space
In a blended space such as the ICE, designers bring together the digital and the physi-
cal to produce a new space. In the case of the ICE the aim was to create a space that
would support collaborative activities such as meetings of various sorts. The concep-
tual space is where people produce meanings and understand where they can go and
what they can do. People need to understand the relationships between the physical
and digital spaces, their organization and structure, so that they can navigate these
spaces. Conceptually these spaces are not easy to understand. It is through, or in, the
conceptual space that people work out roles, task allocation and the social organisa-
tion of their work. This is where they figure out what they need to do and how best
to do it conceptualizing the opportunities and reacting to the physical and digital
constraints.
2.6.4 Designing the Blended Space
By thinking about the three spaces – conceptual, physical and digital – that make up
the spaces of interaction, and the five themes that emerged from our reflections, we
can sensitize designers to the bigger issues. Interaction designers need to design for
space and movement as well as interaction with technology. They need to consider
the devices that people will bring with them and how these can be integrated into
the whole experience of being in, and working in rooms and other interactive spaces.