About
143
Publications
167,872
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
7,287
Citations
Introduction
Skills and Expertise
Publications
Publications (143)
Touch offers important non-verbal possibilities for socioaffective communication. Yet most digital communications lack capabilities regarding exchanging affective tactile messages (tactile emoticons). Additionally, previous studies on tactile emoticons have not capitalised on knowledge about the affective effects of certain mechanoreceptors in the...
Open Access: https://journals.sagepub.com/eprint/P2DRBSJ8PZN5AVBFXYSQ/full
This issue, the first of volume 4, marks the start of Multimodality & Society’s fourth year and provides a good moment to look across the past 3 years to review and reflect on the journal’s contribution to multimodality. Multimodality & Society aims to consolidate and advan...
Touch offers important non-verbal possibilities for socioaffective communication. Yet most digital communications lack capabilities regarding exchanging affective tactile messages (tactile emoticons). Additionally, previous studies on tactile emoticons have not capitalised on knowledge about the affective effects of certain mechanoreceptors in the...
We are in a crisis of touch, prompting a need to reconsider and revalue our tactile behavior and the reciprocity of touching (Kearney, 2021). This crisis provided a starting point for Thresholds of Touch, the performance experiment that underpins this video article. Thresholds was devised through a two- year interdisciplinary collaboration between...
This paper argues that methodological uncertainty, such as that experienced by the social research community through the COVID-19 pandemic (2020–2022) is, and has always been, a vital part of the research landscape. Whilst recognising the many damaging effects of the uncertainties of the pandemic on research and researchers, we home in on the poten...
This paper uses a speculative lens to explore the social and sensory trajectories of Interactive Skin, a class of skin-worn epidermal devices that augment the human body in ways that are significant for affective techno-touch. The paper presents and discusses the use of a speculative narrative on Interactive Skin futures produced through an explora...
Digital devices such as iPads are prevalent in children’s play from an early age. How this shapes young children’s play is an area of considerable debate without any clear consensus on how different forms of play are brought into the iPad interaction. In this study, we examined 98 play activities of children in two preschool settings, featuring 2 a...
This article considers newspapers’ role in shaping the sociotechnical imaginaries of touch, and emerging technologies that digitally mediate touch. It examines the discourses of touch and personal relationships at a distance that circulated in major British broadsheet newspapers during the 2020 outbreak of coronavirus disease-19, alongside dominant...
The social and sensory aspects of touch are critical for human communication, yet the challenges of haptic technology development and a focus on the technological means that digital touch communication often fails to realise the potential and promise of touch. The Manifesto for Digital Social Touch in Crisis responds to this through a call to actio...
This special issue seeks to provoke, challenge, and inspire more multimodal scholars to
engage with and interrogate touch. Collectively the contributions situate touch as part of a
multimodal and multisensorial experience at the intersection of the body, technology and
environment. The contributions offer different routes to critically explore the...
Background
To our knowledge, there has yet to be a comprehensive review of how well hospital services are meeting the needs of children and young people (hereafter referred to as children) with learning disability and their families. The extent to which their experiences differ from those of parents of children without learning disability is not kn...
Collaborative robots are increasingly entering industrial contexts and workflows. These contexts are not just locations for production, they are vibrant social and sensory environments. For better or for worse their entry brings potential to reorganize established tactile and affective dynamics that encompass production processes. There is still mu...
Digitally mediated touch is an emerging and significant area for technology and therefore for design and design education. However, the design of digital touch is a challenge, especially for novice designers, compounded by low awareness and understanding of the sociality of touch and the complexity of communicating felt sensations. This paper prese...
Despite the importance of touch in human–human relations, research in affective tactile practices is in its infancy, lacking in-depth understanding needed to inform the design of remote digital touch communication. This article reports two qualitative studies that explore tactile affective communication in specific social contexts, and the bi-direc...
This paper critically discusses the combination of creative and social research methods to generate a novel approach to explore the multimodal technoscape. This paper draws on an interdisciplinary exploratory case study on interactive skin—an emergent technology that augments and/or interacts with the skin. This paper shows how concepts from skin s...
This paper explores sociotechnical imaginaries for industrial robotics. It is motivated by the prospect of promoting human-centred industrial futures. Investigating the tactility of labour through a critical social perspective the research attends to the future of tactile (tele)robots and elaborates on the concepts of pedagogic, collaborative and s...
This qualitative exploratory research paper presents a Manifesto for Digital Social Touch in Crisis - a provocative call to action to designers, developers and researchers to rethink and reimagine social touch through a deeper engagement with the social and sensory aspects of touch. This call is motivated by concerns that social touch is in a crisi...
This paper describes the development and salience of an original and innovative interdisciplinary approach, Sensoria, that combines methods and techniques from social science and performance to address the methodological challenges of researching sensory/multimodal experiences. It sets out the core components and methodological principles that unde...
This article presents a qualitative analysis of family interaction around digital interactive tabletop exhibits. Parents play a teaching role in museum visits, using strategies from encouragement, to giving directions about using exhibits, to offering explanations that connect an exhibit to children’s previous experiences. Touchscreens and touch ta...
How touch is conceptualised matters in shaping technical advancements, bringing opportunities and challenges for development and design and raising questions for how touch experience is reconfigured. This paper explores the notion of touch in virtual reality (VR). Specifically, it identifies how touch ‘connection’ is realised and conceptualised in...
While interactive touchscreens are currently entering into educational practice, little is known about what this means for learning in early childhood and, in particular, how touchscreens shape action and communication. In this paper, we examine the interactions of 2‐year‐olds and their teachers in a multilingual preschool in Sweden. We analyse the...
“Industry 4.0” marks the advent of a new wave of industrial robotics designed to bring increased automation to “extreme” touch practices and enhance productivity. This article presents an ethnography of touch in two industrial settings using fourth generation industrial robots (a Glass Factory and a Waste Management Center) to critically explore th...
“Industry 4.0” marks the advent of a new wave of industrial robotics designed to bring increased automation to “extreme” touch practices and enhance productivity. This article presents an ethnography of touch in two industrial settings using fourth generation industrial robots (a Glass Factory and a Waste Management Center) to critically explore th...
Touch, often called the ‘first sense’, is fundamental to how we experience and know ourselves, others and the world. Increasingly, touch is being brought into the digital landscape. This paper explores this shifting landscape to understand the ways in which touch is re-mediated in the context of virtual reality. With attention to the sensoriality a...
Bringing touch into VR experiences through haptics is considered increasingly important for user engagement and fostering feelings of presence and immersion, yet few qualitative studies have explored users' iVR touch experiences. This paper takes an embodied approach–bringing attention to the tactile-kinaesthetic body–to explore users' wholistic ex...
This article examines how the use of emergent smart baby monitors re-mediates parent–baby touch, notions of connection, parental sensing and the interpretation of babies’ bodies, and contributes to the formation of subjectivities. Domestic baby monitors are a mid 20th-century phenomenon which normalizes parental anxieties. While baby monitoring is...
This article investigates the remediating effect of bio-sensing technology on touch practices in the context of parent-infant interaction. We examine how the entry of a biosensing technology into the social, sensory and technological ecology of family homes interacts with the ways in which parents and babies know each other and communicate through...
Touch is central to communication and social interaction. For both humans and robots touch is a mode through which they sense the world. A second wave of industrial robots is reshaping how touch operates within the labor process. Recent studies have turned their attention to the role of touch in Human-Robot Interaction (HRI). While these studies ha...
HCI and Industrial Design are both disciplines that are currently experiencing radical transformation in terms of their identity and scope. HCI has moved beyond its origins in human factors and cognitive psychology towards the proactive and generative design of experience. Industrial Design has similarly evolved from a concern with physical form an...
Communication is increasingly moving beyond ‘ways of seeing’ to ‘ways of feeling’. This Open Access book provides social design insights and implications for HCI research and design exploring digitally mediated touch communication. It offers a socially orientated map to help navigate the complex social landscape of digitally mediated touch for comm...
This article makes legible emergent social imaginaries of digital touch for remote communication in personal relationships, with attention to digital touch interfaces. It draws on data from rapid prototyping research workshops with apprentice professionals embedded within digital communication. Touch is discussed with respect to four analytical the...
This chapter examines key ethical considerations and challenges of designing and researching touch technologies, with a focus on incorporating ethical touch sensitivities and values into digital touch communication. We discuss the difficulty of researching and designing ethically in the context of an emerging technological landscape, as reflected i...
This chapter introduces and reflects on the multimodal, sensory and interdisciplinary methodological stance of this book, and the InTouch project more broadly. We introduce our main framework, which combines multimodality and sensory ethnography. We outline the collaborations and interdisciplinary dialogues that we have engaged with to explore digi...
Technologies are intrinsically linked to the ways in which physical, temporal and emotional distances are thought of and managed. Likewise, social relations and communication technologies mutually shape each other as they are developed and maintained. This chapter explores the social connections that digital touch technologies are beginning to shap...
This Chapter explores the potential of the concept of sociotechnical imaginaries for digital touch communication research and design. It defines the social imaginary and discusses how it works to produce and animate shared systems of meaning and belonging that guide and organize the world, in its histories as well as performed visions of desirable...
In this chapter, we make a case for the significance of touch for communication and suggest that developments in sensory digital technologies are bringing touch to the fore in ways that move digital communication beyond ‘ways of seeing’ to include new ‘ways of feeling’. We argue that this shift requires us to take new measure of digitally mediated...
This chapter closes the book with a note on thematic directions, in response to the speculative and emergent character of digital touch communication, signalling our desire and need to keep the conversation open. We point to the significance of a social take on digital touch, particularly with reference to the types of questions this perspective ra...
This chapter provides a descriptive map of digitally mediated touch communication. Whilst acknowledging our everyday interaction with touch screens, our focus is on emergent and semi-speculative touch technologies that want us to be able to touch and feel objects in new ways: from tangibles, wearables, haptics for virtual reality, through to the ta...
This chapter discusses social norms with attention to their significance for researching and designing digital touch communication in a global world, notably gendered and cultural touch norms. It explores how social and cultural norms shape the ways that people (and machines) touch. Touch norms are shaped, regulated and enforced through social, eco...
This paper investigates how family museum visitors crafted learning through interaction with one another and the touch objects of an exhibition. Through a case study of seven families’ interaction, we show how families used touch to bring their interests and resources into dialogue with museum expectations and resources. Using a multimodal approach...
A recent survey reports a decline in disabled children’s services (https://disabledchildrenspartnership.org.uk/wp-content/uploads/2018/06/DCP-survey-report-June-2018.pdf). This presentation will focus on the organisational context in England for healthcare delivery in hospitals to children and young people with learning disabilities. We report staf...
This chapter provides a rationale for a multimodal perspective on meaning, communication and discourse. It draws attention to the range of different modes that people use beyond speech and writing and explores the theoretical and methodological implications of multimodality. The chapter addresses two key questions. First, what is multimodality? Why...
There is a significant gap between technological advancements of digital touch communication devices and social science methodologies for understanding digital touch communication. In response to that gap this article makes a case for bringing the communicational focus of multimodality into dialogue with the experiential focus of sensory ethnograph...
This workshop aims to generate an interdisciplinary research agenda for digital touch communication that effectively integrates technological progress with robust investigations of the social nature and significance of digital touch. State-of-the-art touch-based technologies have the potential to supplement, extend or reconfigure how people communi...
Background:
Children and young people (CYP) with learning disabilities (LD) are a vulnerable population with increased risk of abuse and accidental injury and whose parents have reported concerns about the quality, safety and accessibility of their hospital care. The Care Quality Commission's (CQC) view of best practice for this group of patients...
This chapter sets out a multimodal social semiotic agenda for understanding touch communication practices. It outlines how multimodal social semiotics can provide a framework that can be used to explore the materiality of touch, how these are shaped into touch based semiotic resources and modes, and how their take up by people to communicate is cul...
Multimodality approaches representation and communication as something more than language. It attends to the complex repertoire of semiotic resources and organizational means through which people make meaning – image, speech, gesture, writing, three-dimensional forms, and so on. A social semiotic approach to multimodality sets out to reveal how pro...
Background
There has yet to be a comprehensive review of how well the needs of children and young people (CYP) with Learning Disabilities (LD) are met when accessing hospital services. Hence, we do not know whether existing recommendations, generated in response to evidence of poor experiences and outcomes for adults with LD, can and should be appl...
В сборнике рассматривается мировой и отечественный опыт исследований в области методологической трансдисциплинарности. Обсуждаются и анализируются проблемы «когнитивных оснований» трансдисциплинарных методов в науке как они представлены в современной когнитивистике, философии и методологии науки. В качестве наиболее перспективных трансдисциплинарны...
Introduction
Despite evidence of health inequalities for adults with intellectual disability (ID) there has yet to be a comprehensive review of how well hospital services are meeting the needs of children and young people (CYP) with ID and their families. We do not know how relevant existing recommendations and guidelines are to CYP, whether these...
This paper reports an empirical study that takes a multimodal analytical approach to examine how mobile technologies shape students' exploration and experience of place during a history learning activity in situ. In history education, mobile technologies provide opportunities for authentic experiential learning activities that have the potential to...
The Introduction to the Special Issue is available at: http://vcj.sagepub.com/content/15/3/263.full
The turn to the body in social sciences has intensified the gaze of qualitative research on bodily matters and embodied relations and made the body a significant object of reflection, bringing new focus on and debates around the direction of methodological advances. This article contributes to these debates in three ways: 1) we explore the potentia...
This accessible textbook introduces multimodality: its origins, scope and the potential of multimodal research for understanding the ways in which people communicate. The authors illustrate the key concepts and methods in various domains while demonstrating how to engage critically with the notion of multimodality. Readers will learn to recognize s...
In this paper we examine methodological innovation in the social sciences through a focus on researching the body in digital environments. There are two strands to our argument as to why this is a useful site to explore methodological innovation in the social sciences. First, researching the body in digital environments places new methodological de...
This chapter gives an introduction to social semiotics, its conceptual origins in linguistics, notably in the work of Halliday, and how it has been developed as an approach within multimodality. The key principles of this approach are discussed, and five key concepts are outlined. The chapter illustrates the application of a social semiotic multimo...
This paper explores how to research the opportunities for emotional engagement that mobile technologies provide for the design and enactment of learning environments. In the context of mobile technologies that foster location based linking, we make the case for the centrality of in-situ real-time observational research on how emotional engagement u...
Museums increasingly use digital technology to enhance exhibition experiences for families, notably in relation to physically
mediated installations for young children through natural user interfaces. Yet little is known about how families and children
engage with such installations and the kinds of interactive experiences they engender in museum s...
The increased acquisition of touch-screen technologies, such as tablet computers, in both homes and schools raises important questions about their role for very young children’s learning and development. Their inherent touch-based interaction offers new opportunities for mark making practices, which are linked to literacy development, through the e...
This paper takes a multimodal approach to analysing embodied interaction and discourses of scientific investigation using an interactive tangible tabletop. It argues that embodied forms of interaction are central to science inquiry. More specifically, the paper examines the role of hand actions in the development of descriptions and explanations of...
The aim of the study was to explore young children's touch-based interaction with Tablets. In particular it aimed to examine how finger painting might change in this digital context versus the physical context, and whether it engenders different kinds of touch movements or changes the nature of the painting process. Children aged between 27 and 37...
This paper describes the application of the visual social semiotics schema developed by Kress and van Leeuwen (1996) to sexual health leaflets and posters for young people (age 13 to 19 years) in order to explore how male heterosexuality is managed at a visual level. It illustrates the use of the method, and its advantages and drawbacks in comparis...
Touch-based interaction is increasingly a key feature of digital learning environments, yet we
know little about the specific ways in which digitally mediated touch reshapes interaction for
very young children. This paper examines how finger painting processes, a common activity in
early years learning environments, might change in digital (iPad) v...
The third wave in HCI reveals how embodiment matters in post-WIMP computing systems. Yet it is still unclear what methods provide effective insight into the nature of embodiment in HCI in relation to both design and use. This paper presents work in progress on MIDAS, a cross-disciplinary methodological research project on embodiment and technology...
A multimodal semiotic approach is applied in this chapter to three examples to illustrate how the use of digital technology in museums and galleries can re-mediated the visitor experience (Jewitt, 2009; Kress, 2009). The examples are selected to expand upon the themes raised in Chapter X. They each explore different technologies, contexts and purpo...
Enhanced capabilities of modern smartphones offer the potential to design tools that support new forms of teaching and learning. Increased access to mobile‐networked environments and geospatial systems provide opportunities for developing new educational experiences that support a geospatial approach to science, fostering new ways of thinking about...
The MODE project aims to provide systematic ways to investigate all modes of communication within digital environments, whether they are sites of learning, work, or social networking sites. It is developing multimodal methodologies that look beyond language at other forms of communication and interaction (e.g. visual, auditory, and bodily) to addre...
This project addresses a pressing problem for contemporary research: how to synthesise approaches from the arts and social sciences to develop innovative methods of research. IOE Research Briefings are short descriptions of significant research findings, based on the wide range of projects carried out by IOE researchers.
The methods of data collection that we choose determine the kinds of data that we have access to, and thus shape analyses. In the context of novel interfaces where different modes, available through the environment and context, mediate the interaction, understanding methodological approaches is critical. This paper examines alternative methods of d...
Tangible and multitouch technologies offer new opportunities for physically interacting with objects and digital representations, foregrounding the role of the body in interaction and learning. Developing effective methodologies for examining real time cognition and action, and the relationship between embodiment, interaction and learning is challe...
Research on and with digital technologies is everywhere today. This timely, authoritative Handbook explores the issues of rapid technological development, social change, and the ubiquity of computing technologies which have become an integrated part of people's everyday lives. This is a comprehensive, up-to-date resource for the twenty-first centur...
This chapter sets out to investigate and better understand the consequences of digital technologies as a social cultural tool for making, mobilising and circulating texts in the classroom and re-mediating classroom interaction and practices for learning. It explores how the use of digital technologies in the school reconfigures and redesigns the pe...
Multimodal perspectives on teaching and learning build on the basic assumption that meanings are made (as well as distributed, interpreted, and remade) through many forms and resources of which language is but one—image, gesture, gaze, body posture, sound, writing, music, speech, and so on (Kress & van Leeuwen, 2001; Jewitt, 2009).Keywords:language...
With the recent explosion of technology into the world of education across the globe, this book sets out a framework for rethinking the three key areas of schooling that are most affected by technology's impact on education today: knowledge as curriculum; learning and pedagogy and literacy across the curriculum. A well-known author in this field, J...
In this paper we exemplify how a social semiotic approach to pupils’ multimodal texts (texts which draw on and make available to the senses a range of resources, including the visual, material, and actional) can provide a way into understanding learning. We suggest that learning can be seen as a transformative process of sign making. Specifically,...
The construction of entities (such as cells) is a gradual process, not the result of direct and immediate reading of the ‘facts.’
This paper argues that this is the case for both scientists and pupils in science education. The paper is based on an analysis
of the historical formation of the notion of cells by scientists, and our analysis of a lesso...
The aim of this paper is to show how a substantive area of social research –learning– can be investigated using a multimodal social semiotic approach. We apply the approach to three different institutions – a school, a museum and a hospital, illustrating key concepts and addressing issues around pedagogy and technology in contemporary society. A mu...