Fig 4 - available via license: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International
Content may be subject to copyright.
The 2D matrix of the number of agents (persons ≥ 2 and robots) in HRI interaction extracted from the literature review of Sebo et al. (2020). Examples of the names of the robots used are placed on orange tiles with the count of overall studies identified by the researchers (refer to Table 1 in Sebo et al. (2020) for further details).
Source publication
Our daily human life is filled with a myriad of joint action moments, be it children playing, adults working together (i.e., team sports), or strangers navigating through a crowd. Joint action brings individuals (and embodiment of their emotions) together, in space and in time. Yet little is known about how individual emotions propagate through emb...
Contexts in source publication
Context 1
... an early intervention report, ASD children who received structured intervention focused on imitation and joint attention improved their social interaction skills, such as gaze following and requesting ( Warreyn and Roeyers, 2014). Similarly, In sum, research in ASD and SCZ (both socio-motor interaction deficits) shows complex relationship between multi-faceted difficulties (such as differentiation from others, poor synchrony and imitation) and the ability to act together with others and understand their emotions (see Figure 4). ...
Context 2
... despite being 'emotionally neutral', in a study by Kochigami et al. (2018) robots NAO and PEPPER played social roles by creating social ties between human group members (children and adults) and successfully facilitating interaction between them. Examples of similar studies are limited in number (see Figure 4). Sebo et al. (2020) pinpointed key messages emerging from the current state-of-the-art: (i) behavior in one person to one robot does not interpolate on the group behavior; (ii) verbal and non-verbal robot behavior shapes the response within the group and can support cohesion; (iii) people are more likely to engage with a robot when they are in groups; (iv) similarity (anthropomorphism) to humans plays a role in the integration of robots in a group. ...
Citations
... Motor components of social interactions have attracted increasing interest for several decades, but only recently multidisciplinary collaborations have unravelled their contributions to social interactions [2,9]. Movement is indeed a core aspect of social interactions for achieving mutual understanding and social connection through (i) speech/gesture coordination, (ii) multi-level movement qualities and (iii) synchrony and mimicry. ...
... Furthermore, movements also participate in reading others' minds by providing additional cues for inferring others' intentions in non-verbal and daily interactions [6]. Finally, the spatiotemporal matching of movements (i.e., synchrony/imitation/mimicry; see [9] for an overview) can also participate in fostering social connectedness and fulfilling a sense of belongingness [52]. Consequently, social interactions without motion are challenging, and computer-mediated communication (CMC) often disrupt the natural flow of social interactions [12]. ...
... In fact, JA is a process of "executive" perception and recognition of others' intentions and extracting meaningful information from others' behavior. For example, individuals ensure effective synchronization with participants who share a common goal by inferring shared intentions from perceived information and subsequently adapting to each behavioral plan (Takagi et al., 2019;Bieńkiewicz et al., 2021). Theoretical models of joint attention suggest that JA is closely related to EF, particularly inhibition and switch (Morales et al., 2000); different types of JA reflect unique aspects of executive functioning. ...
Joint attention (JA) is fundamental to the development of children's social functioning; both its response and initiation are closely related to executive function (EF), but the relationship between JA and EF has been relatively rarely studied. The present study aimed to investigate the between-condition differences in brain activation and synchronization of JA under four conditions: (1) stranger-Initiating Joint Attention (Stranger-IJA); (2) teacher-Initiating Joint Attention (Teacher-IJA); (3) stranger-Responding to Joint Attention (Stranger-RJA); and (4) teacher-Responding to Joint Attention (Teacher-RJA). It also aimed to explore the relationships between neuroimaging data and children's inhibitory control levels. To address these two goals, the present study employed 41 (aged 58.61 ± 8.64 months, 24 boys) preschool children through behavioral and functional Near-Infrared Spectroscopy (fNIRS) brain imaging assessment to measure children's EF and brain function under JA, respectively. The results revealed that: (1) a significantly higher prefrontal cortex (PFC) activation was triggered in IJA than RJA; (2) a significantly higher brain activation was triggered in JA with a stranger than with a teacher; (3) a significantly higher index of synchronization asymmetry was evoked in the left and right PFC during interaction with the teacher than with the stranger; and (4) preschoolers' brain activation and synchronization were correlated with their inhibitory control level. The findings advance our understanding of preschoolers' social cognitive development with a biological aspect, offer an opportunity to understand the potential risk of the neural disorder in preschoolers, and provide a basis and insight for preventing neural developmental disorders.
... Moreover, joint attention is a process of "executing" the perception and recognition of others' intentions and extracting critical information from the action of others. On the one hand, individuals ensure effective synchronization with participants who share a common goal by inferring shared intentions and subsequent adaptations to each behavioral plan from perceived information (Bieńkiewicz et al., 2021;Takagi et al., 2019). On the other hand, RJA is primarily controlled by temporal and parietal systems involved in dissociation and orientation of attention, as well as monitoring the behavior of others (Mundy & Neal, 2000). ...
... Joint attention is crucial for developing emotions, resilience, executive function, learning, and other abilities in preschool years. The above literature review has demonstrated the importance of joint attention and a strong link between emotion and joint attention (Bieńkiewicz et al., 2021;Ornaghi et al., 2014). However, these studies have focused on adults or adolescents, and fewer neuroimaging studies have explored different emotion processing of preschoolers with the interacting partner. ...
Emotions and joint attention are highly associated and mutually influenced during preschool, the critical period for early emotional and cognitive develop- ment. However, few studies have explored the neuropsychological mechanism of joint attention with preschoolers and their partners under different emotions. This study has examined the prefrontal activation under a comprehensive emo- tional joint attention task in 45 preschoolers (25 boys, Mage = 58 ± 9.02 months) to compare the different influences of partners' positive, neutral, and negative emo- tions. Analysis of the functional near-infrared spectroscopy data indicated that the participants' prefrontal activation triggered by joint attention in positive and negative emotions was significantly higher than in neutral emotions. Moreover, their brain synchronization intensity was significantly higher in positive emo- tions of joint attention than in negative emotions. These findings advance our understanding of the neural mechanism of early childhood emotional processing under joint attention and provide a neural perspective to explain the effects of different emotions on preschoolers' social cognition.
... Repeated and reinforced mirroring by parental figures allows children to develop a sense of 'self ' (identity), and learn how to regulate emotional states on their own (restore allostasis). In adulthood, signalling emotions (e.g., friendliness) and reading the state of others, supported by societal reinforcement (feelings of affiliation 4,5 ) and hormones such as oxytocin 6,7 , feeds into our cognitive expectations of what is going to happen in our direct environment in the nearest future (i.e., threat). And yet, our understanding of how that happens in a heartbeat, across different modalities (such as movement) that might convey information about emotions and intentions 8 , has largely escaped the modern interdisciplinary science. ...
... Pioneering work on joint drumming showed that physiological and behavioural synchrony was predictive for the perceived cohesion among participants 14 . Links between physiological synchrony and cohesion have also been reported in crowds during live sport event viewing 15 , and group decision making tasks 16 , cementing the stance that social synchronisation is of paramount importance for relational emotions (i.e., affiliation, likability, admiration or attractiveness) in the domains of sport, art and education 10 , with profound social consequences 4,5 . ...
... Notably to date, a plethora of emotion and movement research conducted experiments based on ' enacted' emotions i.e. 21 , which do carry hallmarks of actual emotional expression, but are often exaggerated (especially if actors are the experimental subjects) 4 . And while there are other studies that involved autobiographical recall 22 , or induction with pictorial/video stimuli in psycho-physical context, but without movement data 23 , these methods are weak in terms of induction to produce a pronounced emotional arousal. ...
The ability to synchronise with other people is a core socio-motor competence acquired during human development. In this study we aimed to understand the impact of individual emotional arousal on joint action performance. We asked 15 mixed-gender groups (of 4 individuals each) to participate in a digital, four-way movement synchronisation task. Participants shared the same physical space,
but could not see each other during the task. In each trial run, every participant was induced with
an emotion-laden acoustic stimulus (pre-selected from the second version of International Affective Digitized Sounds). Our data demonstrated that the human ability to synchronise is overall robust to fluctuations in individual emotional arousal, but performance varies in quality and movement speed as a result of valence of emotional induction (both on the individual and group level). We found that three
negative inductions per group per trial led to a drop in overall group synchronisation performance (measured as the median and standard deviation of Kuramoto’s order parameter—an index measuring the strength of synchrony between oscillators, in this study, players) in the 15 sec post-induction. We report that negatively-valenced inductions led to slower oscillations, whilst positive induction afforded faster oscillations. On the individual level of synchronisation performance we found an effect of empathetic disposition (higher competence linked to better performance during the negative induction condition) and of participant’s sex (males displayed better synchronisation performance with others). We believe this work is a blueprint for exploring the frontiers of inextricably bound worlds of emotion and joint action, be it physical or digital.
... It was subsequently integrated into the GAMA simulation platform. Then they demonstrated how to use BEN to simulate the evacuation of a burning nightclub [34] Bridging the gap between emotion and joint action Marta M.N. Bieńkiewicz et al made a review about the gap between emotion and joint action they first identified the gap and then stockpile evidence showing the strong entanglement between emotion and acting together from various branches of sciences then they proposed their integrative approach to bridge the gap and link them in behavioural neuroscience and digital sciences. ...
Nowadays, we are dealing with panic and unpleasant situations in which, we are constrained to make crucial decisions in a limited delay, due to the mixed emotions that may affect our decision, especially FEAR, this kind of emotion occurs when unwanted or uncontrollable events are present in the environment. These recent years, fear modelling has been well researched and since this emotion is usually associated with the fact that one or more fundamental desires are at stake Unluckily, most of these models miss that FEAR does not always occur similarly in all agents. This paper proposes a new conceptual architecture with a new component by extending BDI logic with the emotion of FEAR, so that the new Emotional-BDI agents may better cope with extremely dynamic unpleasant situations in their surroundings. We also address how we verify the emotional properties by employing a model checker NuSMV. The proposed architecture confirms that NuSMV can be applied to verify the emotional specifications we can program agents that are capable of reasoning over emotions, our experimental results indicate the viability and efficiency of our model. Keywords: Emotional-BDI, Model checking, NuSMV, CUDD, Unpleasant situations.
... In a variety of sports, such as basketball free-throw shooting, rugby side-stepping (11), and fly ball catching, the use of virtual reality as a training simulator has shown varying degrees of effectiveness. The task used by Bieńkiewicz et al. (12) was free throw shooting in basketball. First, seasoned athletes fired on a virtual simulator, averaging 47.1% accuracy with terminal feedback after each shot. ...
Metaverse sports arena is gaining popularity globally that empowers virtual reality sporting experience through digital avatars. The main objective of the current study is to explore the impact of the Metaverse-based virtual reality sporting experience on the endurance performance of young Chinese athletes, with the mediating role of their mental health condition and performance anxiety. The study's participants mainly included Chinese athletes, especially the sample group is an accurate depiction of young athletes using a convenience sampling approach. SEM-AMOS statistical software was used for the analysis and validation of the proposed relationships. The study findings statistically validate that mental health and performance anxiety fully mediate the direct associations between virtual reality sporting experiences and the endurance performance of young Chinese athletes. Interestingly, the mental health condition of the young Chinese athletes imposes a greater impact on their endurance performance, in contrast to the adverse effects of their performance anxiety. The outcomes of the present research guide young athletes on the opportunities to enhance their virtual reality sporting abilities and boost their endurance performance. Policymakers can also build systems to dissolve physical and geographical barriers, reduce performance anxiety, and sustain mental health in virtual reality sporting events through the metaverse.
... In a recent review (Bieńkiewicz et al., 2021), we emphasised the need for reconciliation between emotion and joint action research. In line with this proposal, this study aims to show the effects of experimentally induced emotional valence (positive emotion vs. neutral state vs. negative emotion) on group motor synchronisation (arm movements between three people). ...
Emotions are a natural vector for acting together with others and are witnessed in human behaviour, perception and body functions. For this reason, studies of human-to-human interaction, such as multi-person motor synchronisation, are a perfect setting to disentangle the linkage of emotion with socio-motor interaction. And yet, the majority of joint action studies aiming at understanding the impact of emotions on multi-person performance resort to enacted emotions, the ones that are emulated based on the previous experience of such emotions, and almost exclusively focus on dyadic interaction. In addition, tasks chosen to study emotion in joint action are frequently characterised by a reduced number of physical dimensions to gain experimental control and subsequent facilitation in data analysis. Therefore, it is not clear how naturalistically induced emotions diffuse in more ecological interactions with other people and how emotions affect the process of interpersonal synchronisation. Here, we show that positive and negative emotions differently alter spontaneous human synchronous behaviour during a multi-person improvisation task. The study involved 39 participants organised in triads who self-reported liking improvisational activities (e.g., dancing). The task involved producing improvisational movements with the right hand. Participants were emotionally induced by manipulated social feedback involving a personal ranking score. Three-dimensional spatio-temporal data and cardiac activity were extracted and transformed into oscillatory signals (phases) to compute behavioural and physiological synchrony. Our results demonstrate that individuals induced with positive emotions, as opposed to negative emotions or a neutral state, maintained behavioural synchrony with other group members for a longer period of time. These findings contribute to the emerging shift of neuroscience of emotion and affective sciences towards the environment of social significance where emotions appear the most-in interaction with others. Our study showcases a method of quantification of synchrony in an improvisational and interactive task based on a well-established Kuramoto model.
... Many hyperscanning studies of social interactions involving auditory communication have been concerned with the synchrony within or between brains during cooperative or joint action (for reviews, see Keller et al., 2014;Bieńkiewicz et al., 2021;Kelsen et al., 2022). For instance, Konvalinka et al. (2014) used electroencephalographic (EEG) hyperscanning to examine alpha (8-12 Hz) and beta (13-30 Hz) activity during auditory interactions in the form of index finger tapping. ...
Auditory communication is an essential form of human social interaction. However, the intra-brain cortical-oscillatory drivers of auditory communication exchange remain relatively unexplored. We used improvisational music performance to simulate and capture the creativity and turn-taking dynamics of natural auditory communication. Using magnetoencephalography (MEG) hyperscanning in musicians, we targeted brain activity during periods of music communication imagery, and separately analyzed theta (5–7 Hz), alpha (8–13 Hz), and beta (15–29 Hz) source-level activity using a within-subjects, two-factor approach which considered the assigned social role of the subject (leader or follower) and whether communication responses were improvisational (yes or no). Theta activity related to improvisational communication and social role significantly interacted in the left isthmus cingulate cortex. Social role was furthermore differentiated by pronounced occipital alpha and beta amplitude increases suggestive of working memory retention engagement in Followers but not Leaders. The results offer compelling evidence for both musical and social neuroscience that the cognitive strategies, and correspondingly the memory and attention-associated oscillatory brain activities of interlocutors during communication differs according to their social role/hierarchy, thereby indicating that social role/hierarchy needs to be controlled for in social neuroscience research.
... By the point of completing this thesis, the work presented in this chapter has received 6 citations (excluding the self-reference by my own works). In a review about analyzing the gap between emotion and joint action from the perspective of behavioral neuroscience [131], our work was taken as an example to show how advanced machine learning technique has directly incorporated the learning of contextual information to aid the detection of affect-related behavior detection. ...
Chronic pain (CP) rehabilitation extends beyond physiotherapist-directed clinical sessions and primarily functions in people's everyday lives. Unfortunately, self-directed rehabilitation is difficult because patients need to deal with both their pain and the mental barriers that pain imposes on routine functional activities. Physiotherapists adjust patients' exercise plans and advice in clinical sessions based on the amount of protective behavior (i.e., a sign of anxiety about movement) displayed by the patient. The goal of such modifications is to assist patients in overcoming their fears and maintaining physical functioning. Unfortunately, physiotherapists' support is absent during self-directed rehabilitation or also called self-management that people conduct in their daily life. To be effective, technology for chronic-pain self-management should be able to detect protective behavior to facilitate personalized support. Thereon, this thesis addresses the key challenges of ubiquitous automatic protective behavior detection (PBD). Our investigation takes advantage of an available dataset (EmoPain) containing movement and muscle activity data of healthy people and people with CP engaged in typical everyday activities. To begin, we examine the data augmentation methods and segmentation parameters using various vanilla neural networks in order to enable activity-independent PBD within pre-segmented activity instances. Second, by incorporating temporal and bodily attention mechanisms, we improve PBD performance and support theoretical/clinical understanding of protective behavior that the attention of a person with CP shifts between body parts perceived as risky during feared movements. Third, we use human activity recognition (HAR) to improve continuous PBD in data of various activity types. The approaches proposed above are validated against the ground truth established by majority voting from expert annotators. Unfortunately, using such majority-voted ground truth causes information loss, whereas direct learning from all annotators is vulnerable to noise from disagreements. As the final study, we improve the learning from multiple annotators by leveraging the agreement information for regularization.
... ACM Digital Library Author keyword contains: "database", "corpus", or "dataset" AND Anywhere in the full text includes: "action", "activity", "motion", or "movement" 2,000 (cap) 14 6 Oct 2020 IEEE Xplore Digital Library Author keyword contains: "database", "corpus", or "dataset" AND Anywhere in the full text includes: "action", "activity", "motion", or "movement" 1,228 182 28-29 October 2020 ...
... paucity of (open) data on social scenarios in our survey highlights the domination of 'first-person' behavioural sciences (neurosciences, cognitive sciences, or biomechanics) [14,73]. ...
Movement dataset reviews exist but are limited in coverage, both in terms of size and research discipline. While topic-specific reviews clearly have their merit, it is critical to have a comprehensive overview based on a systematic survey across disciplines. This enables higher visibility of datasets available to the research communities and can foster interdisciplinary collaborations. We present a catalogue of 704 open datasets described by 10 variables that can be valuable to researchers searching for secondary data: name and reference, creation purpose, data type, annotations, source, population groups, ordinal size of people captured simultaneously, URL, motion capture sensor, and funders. The catalogue is available in the supplementary materials. We provide an analysis of the datasets and further review them under the themes of human diversity, ecological validity, and data recorded. The resulting 12-dimension framework can guide researchers in planning the creation of open movement datasets. This work has been the interdisciplinary effort of researchers across affective computing, clinical psychology, disability innovation, ethnomusicology, human-computer interaction, machine learning, music cognition, music computing, and movement neuroscience.
