Conference Paper

Happiness counter: Enhancing positive mood and communication with smile-encouraging digital appliances

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

William James, the noted psychologist and philosopher, believed that smiling has a positive effect on our mind. James' view, which was confirmed by several psychological studies, was that we become happier when we laugh. In this paper, we propose a new digital appliance that encourages the act of smiling in our daily lives. This system is designed for people who may not always realize when they are in low spirits and/or have difficulty with smiling. In addition, we believe that this system will foster casual conversation and prompt communications with other people. Our appliance, called the HappinessCounter, combines visual smile recognition, user feedback, and network communication. We conducted two trials of the HappinessCounter system, the first with a single occupant and the second with a couple living together. The system had positive effects on user's mood and prompted communication among family members, thereby increasing their positive mood as well.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... However, the facial interface is not generally supported to be manipulated by an external user. Researchers have challenged to deform a human face by graphical approaches [310], mechanical approaches [186] and physiological approaches [314] in order to apply the results for graphical usage and to explore the effect to emotion. ...
... As like prior researchers have attempted to enhance communication by modifying facial expressions with image processing in video chats [295], a similar approach can be taken in the real-world faceto-face communication. One can get used or train to create a facial expression [314] in a more passive way. Sometimes people may enjoy negative emotions (e.g., fear) [5], therefore, these kinds of experience can be augmented by simulating or enhancing negative emotions. ...
... It is known that training or activating the facial muscles is valid to effect emotions [314,48], therefore, usage of EMS on the face and to stimulating the facial muscles may also be effective. Tsujita et al. [314] designed a system to enhance positive emotions and to increase happiness. ...
Thesis
Full-text available
The border of computer technology and human beings have become ambiguous, where they are integrated and merged with each other. The human body is being modified and even artificial components are implanted inside the body. However, we often face fear or anxiety in such approaches, due to the unclearness of safety and threat of social acceptance. There has been an interest in using surface electrodes and electricity to modify the body without direct implants to the body. I consider this approach to be positioned between wearables and implants, where artificial electrical signals are applied inside the body from the outside surface. In this thesis, I call this approach “Electrical Body Hacks.” This thesis aims to bridge the gap between wearables and cyborgs (implants), by practice through electrical body hacks, and to increase the acceptance of such attempts by developing an experimental platform. However, electricity is not completely safe for the human body. The knowledge required for such consideration is spread throughout a large number of research fields. Therefore, I start by reviewing previous research pertaining to HCI in which users come into contact with electricity, as well as safety consideration and guidelines. As a practice for my approach, we present a multi-channel electrical muscle stimulation (EMS) toolkit, in order to ease the access of users for electrical body hacks. We organized a workshop and found that multi-channel EMS has a significant demand for human augmentation purposes. To explore cognitive understandings of EMS on an unacceptable body part, I present studies of EMS applied on a human face. EMS may stimulate the face to express emotions through the facial muscle actuation, which may work better for negative emotions. Furthermore, a user study was performed to explore the effect of our approaches with combinations with virtual reality (VR) contents, which was found effective to enhance the experience. Finally, we present an approach which allows a human body to activate low-power electronic devices by touching them, and to present application domains that overlap with wearable and cyborg approaches. Consideration for the acceptability of such systems are key issues, and discussions are required. Therefore, I present a careful discussion to address this issue, and to help increase the acceptability of the research area and future work. I envision future development and research of body modification to be more safe and acceptable for human augmentation.
... The CFO theory has also been linked to affect and stress. Nonverbal cues such as smiles and laughs are easily contagious and prone to mimicry during in-person or telephone communications (Hess & Fischer, 2013), which can increase one's positive mood and reduce one's negative mood (Tsujita & Rekimoto, 2011). The social and physical presence of others can also provide more support for individuals under stress (Holtzman et al, 2017;Jakubiak & Feeney, 2016). ...
... Furthermore, higher previous-day stress exposure was related to less telephone calling on the subsequent day. These findings are consistent with past studies and the CFO theory, which suggest that the lack of nonverbal, verbal, physical, and social cues in text-based CMC can more easily induce misinterpretation in one's message, whereas communicating with voice can better allow one to accurately express oneself and assess the receiver's reaction (Derks et al., 2008;Holtzman et al., 2017;Tsujita & Rekimoto, 2011). Text-based CMC can provide a way of communicating with others in an efficient and timely manner, but it is important to note that using text-based CMC frequently for social interactions with family and friends is associated with greater stress exposure on both the concurrent and subsequent day. ...
Article
Full-text available
Communication with one's social network can take place in person or using technology. Past studies have mainly focused on the effects of communication modality (in-person, telephone calling, text messaging, and internet) on stress and affect at a between-person level by exploring the individual differences. Yet few studies have compared such effects at a within-person level, that is, how an individual varies over time. We conducted a diary study over 7 days for 145 participants (ages 22-94) mostly from the greater Boston area to test the role each communication mode played in daily stress exposure, stress reactivity, and positive and negative affect using within-person analyses. Multilevel modeling results revealed that days with more frequent text messaging were associated with greater stress exposure and negative affect. Days with more in-person communication were associated with more positive affect. Days with more telephone calls were associated with less negative affect. Internet communication was not associated with stress or affect at a within-person level. To address the directionality of our findings, we also conducted lagged analyses that suggested that higher previous-day frequency of text messaging was related to higher stress exposure on the subsequent day. In addition, higher previous stress exposure was related to less telephone calling on the subsequent day. Implications and future research are discussed with a focus on how social interactions via different communication modes with one's social network can make a difference for daily well-being.
... Apart from these reactive approaches, prophylaxis of negative states could also be interesting to investigate. We can learn from experiments in other domains, for example, Tsujita and Rekimoto's smart home appliance which achieved positive influences on emotions by making users smile [49], or we consider gamification approaches in automotive environments, which try to bring joy and driver safety together by focusing the driver's attention to the driving task in a playful manner [43,46]. One current challenge in automotive research is bringing user experience and safety together, as measures to make the driver more aware of their current state might raise reactance when users feel patronized by the system [9]. ...
... Fun levels decreased over time if no positive expressions were detected. The general idea stems from approaches of increasing driver engagement through gamification [45] and the notion that the act of smiling itself makes people feel happier [49]. In our prototype the values for driver health was simulated with random positive values as we did not collect health data during the experiment. ...
Chapter
Full-text available
Driver state detection is an emerging topic for automotive user interfaces. Motivated by the trend of self-tracking, one crucial question within this field is how or whether detected states should be displayed. In this work we investigate the impact of demographics and personality traits on the user experience of driver state visualizations. 328 participants experienced three concepts visualizing their current state in a publicly installed driving simulator. Driver age, experience, and personality traits were shown to have impact on visualization preferences. While a continuous display was generally preferred, older respondents and drivers with little experience favored a system with less visual elements. Extroverted participants were more open towards interventions. Our findings lead us to believe that, while users are generally open to driver state detection, its visualization should be adapted to age, driving experience, and personality. This work is meant to support professionals and researchers designing affective in-car information systems.
... Considering the beneficial influence smiling can have on overall affective state, a number of systems developed by the HCI community focus on encouraging smiles. Tsujita et al. [61] use household cameras to detect smiles and provide feedback to users in order to increase self-awareness about smiling behavior and mood. Hernandez et al. [29] station cameras on a college campus to detect smiles in real time and reflect upon the community's emotional reactions. ...
... We find that this baseline performs slightly worse with a mean absolute error of 3.55. L = 7 L = 10 L = 14 k = 7 3.70 3.95 4.19 k = 13 3. 53 3.68 4.10 k = 10 3. 61 3.75 4.05 Table 6. Mean Absolute Error (MAE) for rolling forecast of economic confidence (L = # of days in the future we forecast, k = # of days of historical data used to train model) ...
Conference Paper
The increasing adoption of social media provides unprecedented opportunities to gain insight into human nature at vastly broader scales. Regarding the study of population-wide sentiment, prior research commonly focuses on text-based analyses and ignores a treasure trove of sentiment-laden content: images. In this paper, we make methodological and computational contributions by introducing the Smile Index as a formalized measure of societal happiness. Detecting smiles in 9 million geo-located tweets over 16 months, we validate our Smile Index against both text-based techniques and self-reported happiness. We further make observational contributions by applying our metric to explore temporal trends in sentiment, relate public mood to societal events, and predict economic indicators. Reflecting upon the innate, language-independent aspects of facial expressions, we recommend future improvements and applications to enable robust, global-level analyses. We conclude with implications for researchers studying and facilitating the expression of collective emotion through socio-technical systems.
... For example, Tsujita and Rekimoto presented a variety of digital appliances, such as refrigerators, alarm clocks, mirrors et cetera that require the user to smile in order to function properly [20]. In [19], they described two field tests that they conducted with the refrigerator. This refrigerator integrated a camera to recognize and count the user's smiles, and only when the user smiled, the system facilitated opening the refrigerator door. ...
... In this paper, we presented a smartphone application integrating two features of positive technology: a Social/Interpersonal interface that aims at introducing a part of kinesics, i.e., facial expressions, in the computer-mediated communication of emotions. Moreover, this prototype takes advantage of the recorded emotions shared by the user in order to provide some statistics to enable self-reflection about the shared emotions; this aims at improving mental health and if the user did not smile enough during the day, a persuasive interface inspired by Tsujita and Rekimoto's work [19], motivates the user to smile. ...
Conference Paper
In this paper, we describe a smartphone application that aims at motivating users to use facial expressions. This has a twofold goal: to reintroduce the use of facial expressions as nonverbal means in the computer-mediated communication of emotions and to provide the opportunity for self-reflection about the personal emotional states while fostering smiles in order to improve mental wellbeing. This paper provides a description of the developed prototype and reports the results of a first observation study conducted during an interactive event.
... In recent years, there has been tremendous progress in technologies for analyzing and synthesizing video of human faces, e.g., [3,40,49,50,60], with new applications in smart home technologies [54], health [12,22], and other felds. ...
... One study found that a smile could prevent stress [5] and lead to healthy mental wellbeing. Another study by Tsujita et al. [6] found that performing smiles in a daily fashion can enhance the positive mood. ...
Conference Paper
Full-text available
This paper presents a video game system with a smile interface designed to promote the player experience. We introduce a new mechanism to an existing endless runner game, “Runner.” Such mechanism allows smiles to be taken as an input to the game system, for triggering boosting period. To detect smiles, a deep-learning facial recognition toolkit named Affdex, by Affectiva, is employed. Evaluation on player experience was done using a shorten version of Game User Experience Satisfaction Scale. Three different game modes are compared: Standard Mode (no smile and boosting), Smile Mode (with smile for boosting), and Auto Mode (auto boosting, without smile). Our results show that the presence of the smile mechanism leads to more enjoyable gameplay.
... For example, they showed a microwave that requires the user to perform a step-aerobics exercise during operation. Another example of the inconvenient system was a refrigerator that contains a smile-awareness sensor so that the user has to smile if they want to open it [23]. These examples were based on the premise that laughing can help people to maintain long-term benefits of health. ...
Article
Full-text available
The latest mobile alarm apps provide wake-up tasks (e.g., solving math problems) to dismiss the alarm, and many users willingly accept such an inconvenience in return for successfully waking up on time. However, there have been no studies that investigate how the wake-up tasks are used and their effects from a human–computer interaction perspective. This study aims to deepen our understanding of how users engage and utilize the task-based alarm app by (1) examining the characteristics of different wake-up tasks and (2) extracting usage factors of hard tasks which involve physical or cognitive task loads over a certain level. We developed and deployed Alarmy, which is a task-based mobile alarm app with four wake-up task features: touching a button, taking a picture, shaking the device, and solving math problems. We collected 42.9 million in situ usage data from 211,273 US users for five months. Their alarm app usage behaviors were measured in two folds: eight alarm-set variables and five alarm-dismiss variables. Our statistical test results reveal the significant differences in alarm usage behaviors depending on the wake-up task, and the multiple regression analysis results show key usage patterns that affect the frequent uses of hard tasks, which are late alarm hours, many snoozes, and relatively more use on weekends. Our study results provide theoretical implications on behavior change as well as practical implications for designing task-based mobile alarm.
... There was reported to exist relationships between physical activity, depression and anxiety symptom, namely, those who do physical exercise frequently tend to have a healthy mental state [6]. In addition, some studies reported that smiling can influence happiness and stress response [7], [8]. Based on these finding, we believe a game system integrating physical exercise and smiling can be an effective mean for promoting simultaneously physical and mental dimensions of well-being, thus we demonstrate such integration in a running game, Runner, in this paper (cf. ...
Conference Paper
Full-text available
This paper presents new features added to Runner, an open-source running game platform designed for games for health research. This game can be played by using body motion through a Kinect device, and the original version of the game focused on physical health promotion. We presented a new mode that aims at enhancing mental well-being through a smile detection mechanism. An integration between motion detection and facial expression detection for a 2D-platform running game, Runner, is presented.
... For example, Choi et al. [7] showed that a computer interface with an emotional display affected people's decision making ability. Detecting and recognizing human emotional state and reacting to emotion change is studied in the field of affective computing [9,10]. This inspired us to use AR to add emotional cues to real objects to evoke human emotional reactions, and also provide real-time information feedback [11]. ...
Conference Paper
Full-text available
In this paper, we explore how Augmented Reality (AR) and anthropomorphism can be used to assign emotions to common physical objects based on their needs. We developed a novel emotional interaction model among personified physical objects so that they could react to other objects by changing virtual facial expressions. To explore the effect of such an emotional interface, we conducted a user study comparing three types of virtual cues shown on the real objects: (1) information only, (2) emotion only and (3) both information and emotional cues. A significant difference was found in task completion time and the quality of work when adding emotional cues to an informational AR-based guiding system. This implies that adding emotion feedback to informational cues may produce better task results than using informational cues alone. © 2018 IEEE.
... FRTs have also been used for emotion detection and have been explored for related everyday practices. For instance, a smile-activated refrigerator detected facial expression, automatically unlocking the refrigerator if a smile was detected, with the hope of encouraging users to smile more [56]. In addition, FRTs have been applied to cosmetics. ...
Conference Paper
With the recent advancement in computer vision, Artificial Intelligence (AI), and mobile technologies, it has become technically feasible for computerized Face Reading Technologies (FRTs) to learn about one's health in everyday settings. However, how to design FRT-based applications for everyday health practices remains unexplored. This paper presents a design study with a technology probe called Faced, a mobile health checkup application based on the facial diagnosis method from Traditional Chinese Medicine (TCM). A field trial of Faced with 10 participants suggests potential usage modes and highlights a number of critical design issues in the use of FRTs for everyday health, including adaptability, practicality, sensitivity, and trustworthiness. We end by discussing design implications to address the unique challenges of fully integrating FRTs into everyday health practices.
... The role of hand gestures has been well studied in communication [31,39] and some work has been done toward capturing the expressiveness of face-to-face communication through both facial (e.g., smiling [69]) and limb (e.g., hugging [41]) interactions. However, research into its utility in emotion-enhanced CMC remains largely lacking [71]. ...
Article
Recent trends in computer-mediated communication (CMC) have not only led to expanded instant messaging through the use of images and videos but have also expanded traditional text messaging with richer content in the form of visual communication markers (VCMs) such as emoticons, emojis, and stickers. VCMs could prevent a potential loss of subtle emotional conversation in CMC, which is delivered by nonverbal cues that convey affective and emotional information. However, as the number of VCMs grows in the selection set, the problem of VCM entry needs to be addressed. Furthermore, conventional means of accessing VCMs continue to rely on input entry methods that are not directly and intimately tied to expressive nonverbal cues. In this work, we aim to address this issue by facilitating the use of an alternative form of VCM entry: hand gestures. To that end, we propose a user-defined hand gesture set that is highly representative of a number of VCMs and a two-stage hand gesture recognition system (trajectory-based, shape-based) that can identify these user-defined hand gestures with an accuracy of 82%. By developing such a system, we aim to allow people using low-bandwidth forms of CMCs to still enjoy their convenient and discreet properties while also allowing them to experience more of the intimacy and expressiveness of higher-bandwidth online communication.
... In order to run, the player has to smile; arching an eye brow results in jumping. Previous research could show that technology which naturally encourages smiling positively affects users' mental state [10,9]. The background of the game uses a live camera view of the current window view so that the player has the impression of running on a guardrail of the street (Figure 1). ...
Conference Paper
Full-text available
Children can be a distraction to the driver during a car ride. With our work, we try to combine the possibility of facial expression recognition in the car with a game for children. The goal is that the parents can focus on the driving task while the child is busy and entertained. We conducted a study with children and parents in a real driving situation. It turned out that children can handle and enjoy games with facial recognition controls, which leads us to the conclusion that face recognition in the car as a entertaining system for children should be developed further to exploit its full potential.
... Despite possibility of increased interruption, micro-interactions with smartwatches provide immense opportunities for timely delivery of behavioral change related information for prompt checking/glancing, which may lead to detailed review/reflection with their smartphones and thereby to durable behavioral changes [17]. Furthermore, smartwatches can be used as ways of enforcing certain behaviors by causing discomfort/inconvenience to users [4,46,50]; e.g., vibrating watches until the users show desirable behaviors. For example, as one of the ways of mitigating smartphone overuse, unlike conventional lock-based self-limiting mechanisms [23][24][25], we can nudge users of overuse by vibrating watches until the users stop using their phones. ...
Article
Smartwaches are the representative wearable or body-worn devices that provide convenient and easy information access. There is a growing body of research work on enabling novel interaction techniques and understanding user experiences of smartwatches. However, there is still lack of user experience research on wearing behaviors of smartwatches, which is critical for wearable device and service design. In this work, we investigate how college students wear smartwatches and what factors affect wearing behaviors by analyzing a longitudinal activity dataset collected from 50 smartwatch users for 203 days. Our results show that there are several temporal usage patterns and distinct groups of usage patterns. The factors affecting wearing behaviors are contextual, nuanced, and multifaceted. Our findings provide diverse design implications for improving wearability of smartwatches and leveraging smartwatches for behavioral changes.
... However, these influences should be aligned with the user's long-term goals (not influencing them in "unwanted ways"). Tsujita and Rekimoto outlined a basic example: We can increase people's happiness by making them smile more [21]. Yoshida et al. proposed an emotion evoking system [22]. ...
Conference Paper
This paper presents a novel smart eyewear that uses embedded photo reflective sensors and machine learning to recognize a wearer's facial expressions in daily life. We leverage the skin deformation when wearers change their facial expressions. With small photo reflective sensors, we measure the proximity between the skin surface on a face and the eyewear frame where 17 sensors are integrated. A Support Vector Machine (SVM) algorithm was applied for the sensor information. The sensors can cover various facial muscle movements and can be integrated into everyday glasses. The main contributions of our work are as follows. (1) The eyewear recognizes eight facial expressions (92.8% accuracy for one time use and 78.1% for use on 3 different days). (2) It is designed and implemented considering social acceptability. The device looks like normal eyewear, so users can wear it anytime, anywhere. (3) Initial field trials in daily life were undertaken. Our work is one of the first attempts to recognize and evaluate a variety of facial expressions in the form of an unobtrusive wearable device.
... Office workers can be grouped based on their office names, and collaborative and competitive activities can be performed (e.g., inter-office pedal count competitions). Another design opportunity is to incorporate inconvenience interactions, in which an interactive mechanism coerces users to make intended actions [54]. For example, pedaling can be required to keep a computer monitor powered on. ...
Conference Paper
Full-text available
Prolonged inactivity in office workers is a well-known contributor to various diseases, such as obesity, diabetes, and cardiovascular dysfunction. In recent years, active workstations that incorporate physical activities such as walking and cycling into the workplace have gained significant popularity, owing to the accessibility of the workouts they offer. While their efficacy is well documented in medical and physiological literature, research regarding the user experience of such systems has rarely been performed, despite its importance for interactive systems design. As a case study, we focus on active workstations that incorporate under desk elliptical trainers, and conduct controlled experiments regarding work performance and a four week long field deployment to explore user experience with 13 participants. We investigate how such workouts influence work performance, when and why workers work out during working hours, and the general feelings of workers regarding usage. Our experimental results indicate that while work performance is not influenced, the cognitive load of tasks critically influences workout decisions. Active workstations were alternatively used as mood enhancers, footrests, and for fidgeting, and there exist unique social and technical aspects to be addressed, such as noise issues and space constraints. Our results provide significant implications for the design of active workstations and interactive workplaces in general.
... The effects were greater when participants viewed themselves in a mirror. Based on this theory, Tsujita and Rekimoto (2011) designed HappinessCounter, a device that recognizes users' smiles, counts the number of smiles and then provides feedback in a mirror. Their field study showed that users became happier and smiled more naturally after ten days. ...
Article
Full-text available
Background: With the increasing quality of smartphone cameras, taking photos has become ubiquitous. This paper investigates how smartphone photography can be leveraged to help individuals increase their positive affect. Methods: Applying findings from positive psychology, we designed and conducted a 4-week study with 41 participants. Participants were instructed to take one photo every day in one of the following three conditions: a selfie photo with a smiling expression, a photo of something that would make oneself happy and a photo of something that would make another person happy. Findings: After 3 weeks, participants' positive affect in all conditions increased. Those who took photos to make others happy became much less aroused. Qualitative results showed that those in the selfie group observed changes in their smile over time; the group taking photos to improve their own affect became more reflective and those taking photos for others found that connecting with family members and friends helped to relieve stress. Conclusions: The findings can offer insights for designers to create systems that enhance emotional well-being.
... In recent years there has been a growing interest in using digital devices to detect and improve mood. For example, the HappinessCounter recognizes smiles, detects frequency of occurrences and measures affect [39]. Due to behaviors related to smiling, participants reported experiencing more positive mood. ...
Conference Paper
Positive wellbeing in the workplace is tied to better health. However, lack of wellbeing in the workplace is a serious problem in the U.S, is rising continually, and can lead to poor health conditions. In this study we investigate factors that might be associated with workplace wellbeing. We report on an in situ study in the workplace of 40 information workers whose mood was tracked for 12 days. We used a mixed-methods study using Fitbit actigraphs to measure sleep and physical activity, computer logging, and repeated daily surveys. We found that sleep and perceived productivity are positively correlated with affect balance (the balance of positive and negative affect), whereas concentration difficulty, and amount of time on workplace email, are negatively correlated with affect balance. Our model explains 48% of the variance of workplace mood. We discuss the value and challenges of multi-faceted measures of health as we move towards designing interdisciplinary digital health research.
... In fact, smiles have become so prevalent that people purposely induce them with standard words and sentences (e.g., "Say cheese!"), and modern commercial cameras automatically detect them to decide when to take the optimal picture. A recent study [19], explored the use of these cameras in household environments to enhance positive mood and prompt communication. Below we will also describe some findings suggesting that smiles elicited by our system momentarily improved mood of the participants. ...
Article
Full-text available
In this study, we created and evaluated a computer vision based system that automatically encouraged, recognized and counted smiles on a college campus. During a ten-week installation, passersby were able to interact with the system at four public locations. The aggregated data was displayed in real time in various intuitive and interactive formats on a public website. We found privacy to be one of the main design constraints, and transparency to be the best strategy to gain participants' acceptance. In a survey (with 300 responses), participants reported that the system made them smile more than they expected, and it made them and others around them feel momentarily better. Quantitative analysis of the interactions revealed periodic patterns (e.g., more smiles during the weekends) and strong correlation with campus events (e.g., fewer smiles during exams, most smiles the day after graduation), reflecting the emotional responses of a large community.
Chapter
Recently, it has been clarified that smiling has a positive effect on both physical and mental health. However, few studies have taught participants how to smile specifically and verified the effects of smiling. This study involved 13 undergraduates and graduate students at A-University practicing the Unpani Exercise in their daily lives for two weeks to clarify the psychological effects of smiling. Before and after the experiment, participants answered the Japanese version of Rosenberg’s Self-esteem Scale, the psychological stress reaction scale, the Japanese version of the Ten Item Personality Inventory, and the mental toughness evaluation scale. To clarify the changes of facial expression by continuing the facial exercises, the participants’ facial expressions were analyzed using the facial expression analysis software FaceReader8 (Noldus Co. Ltd). Participants were instructed to take a picture of their smile before the experiment, on the seventh day of the experiment, and after the experiment. The values of “Happy” calculated from their facial expressions were obtained by analyzing the pictures with FaceReader. The scores in sociality, a factor of the mental toughness scale, increased significantly after the experiment. In addition, the scores of will power and positive degrees showed a marginally significant increase after the experiment. For the other three scales, there was no significant increase in scores. According to the analysis with FaceReader, 8 out of 13 participants showed an increase in their values of “Happy” after the experiment, suggesting that the facial exercise improved their smiles. Therefore, the results suggest that facial exercises are effective in encouraging mental toughness and improving smiling. Based on these findings, facial exercises can be applied to prevent facial muscle weakness caused by masks and activate the brain through facial muscle activity, and they may also be used as a method of mental support.
Article
Smiling is the most common way in which people express positive emotions and happiness. This study proposes Smiley, a smile-recognition smart mirror that can only be seen when users smile at it. To verify if there were emotional changes through smiling, the smile-recognition smart mirror was designed and manufactured for women of childbearing age in their 20s and 30s suffering from premenstrual syndrome (PMS), and a user evaluation was conducted. We recruited four experimental participants and had them use the mirror in their own home for 10 days according to their menstrual cycle, maintain a self-diary, and record smile photos to observe changes in their facial expressions over 10 days. We then carried out in-depth interviews with the users through memory-recall using their smile photo record data and self-diary data organized by date and time, and then qualitatively analyzed the results. Women suffering from PMS continuously experience anger or feel sensitive and depressed. Using Smiley, these negative emotions caused by PMS can be changed to positive feelings and emotional stability by smiling. Moreover, Smiley provides personalized information about the user’s menstrual cycle in color, allowing her to identify and prepare for her cycle and gain psychological stability. The evaluation results demonstrated that smiling evoked positive emotions and positively influenced PMS and users’ daily life. This study is significant in that the positive effects of smiling are not only discussed merely for the sake of the psychological experiment, but have also been used to develop a new technology that can provide users with a wide variety of experiences.
Article
While traditional videoconferencing causes privacy issues, virtual meetings are not yet widely used. Their communication quality still lacks usability and important non-verbal communication cues, such as body language, are underrepresented. We aim at exploring virtual avatars’ body language and how it can be used to indicate meeting attendees’ communication status. By comparing users’ perceptions of avatar behavior, we found that avatar body language across gender can be an indication of communication willingness. We derive resulting body language design recommendations and recommend using attentively behaving avatars as default body language and to indicate being busy through actions of the avatar, such as drinking, typing, or talking on a phone. These actions indicate that users are temporarily busy with another task, but still are attending the meeting. When users are unavailable, their avatars should not be displayed at all and in cases of longer meeting interruptions, the avatar of a user should leave the virtual meeting room.
Chapter
This chapter aims at explaining the mechanisms of persuasion for effective communication in online educational environments. It starts by bringing awareness on the particularities of online educational environments and the psychological prerequisites of online studying. It tackles issues of digital skills as well as emotional intelligence abilities for online teaching, specific mindset and digital readiness, result expectations and assessment. Moreover, principles of microlearning and hybrid learning are explained as major approaches in online education. It then addresses issues related to technology-mediated communication with young ages vs. adults. It makes a brief analysis of mediated communication vs. direct communication in terms of adopted strategies according to age, adaptation of message, and feedback according to the online channel of factors affecting communication in online environments vs. face-to-face communication. Particular attention will be given to digital stress.
Conference Paper
We evaluate the use of Electric Muscle Stimulation (EMS) as a method of amplifying emotional responses to multimedia content. This paper presents an auto-calibration method to stimulate two facial expressions using EMS. We focus on two expressions: frown and smile. We attempted control of facial muscles with facial feedback for automatically calibrating these facial expressions: our computer vision system detects the facial expression and auto-calibrates the EMS parameters (intensity and duration) based on the user's current facial expression. We present results from a pilot study with four participants evaluating the auto-calibration system and collecting initial feedback on the use of EMS to augment, for example, media experiences: while watching movies we can enhance the emotional response of the users during happy and sad scenes by stimulating corresponding face muscles.
Conference Paper
In this paper, we describe a new application that encourages positive emotions. It recognizes facial expressions in real time, and in response to a certain facial expression, the system gives feedback by showing an animation. User's facial expressions drive animations and, animations are used to encourage positive emotions. We conducted an evaluation experiment in which participants used the application and answered a questionnaire. It was found that the application elicited positive emotions such as happiness and surprise. In the future, we plan to develop applications for encouraging positive emotions in various daily situations such as classrooms and conference rooms.
Conference Paper
In this paper, we present a wearable facial expression recognition system that can analyse and enhance a daily experience. Our aim is to create a mindful experience in daily life by connecting the device with everyday objects and service. To this end, we made two prototypes that supports users to keep right side of emotions: 1) a text chatting system that automatically inserts an emoticon based on his/her facial expressions in the end of a comment a user typed, 2) a plant interface controlled by facial expressions. We also analysed multiple users' facial expressions while they played video games. We confirmed that visualization of sensor data from the device shows the possibility for estimating the transition of different facial expressions.
Article
While most traditional user interfaces are intended to pursue "convenience" by eliminating user operations and by typically automating tasks, some new categories of HCI, such as health support, may require explicit human participation and effort to achieve long-term benefits. In these areas, interfaces that require interactions that promote users to perform explicit activities, rather than interfaces that solely perform tasks on behalf of users, are becoming increasingly important. This trend can be a further challenge of interaction design, and we refer it as "inconvenient interactions". In this paper, we discuss why carefully designed inconveniences can enrich our lives, and provide preliminary but concrete examples. We also propose our guidelines for the design of these inconvenient interactions.
Article
Communication skills are essential in our everyday lives. Yet, it can be difficult for people with communication disorders to improve these skills without professional help. Quantifying communication and providing feedback advice in an automated manner would significantly improve that process. Therefore, we aim to propose a method to monitor communication that employs life-logging technology to evaluate parameters related to communication skills. In our study, we measured frequency of smiles as a metric for smooth communication. In addition, smiling can improve happiness even if a smile is mimicked. Ultimately, we provided feedback results to users in a gamified form and investigated the effects of feedback on communication.
Conference Paper
The main goals of this paper involved assessing the efficacy of computer-generated emotion and establishing a method for integrating emotional experience. Human internal processing mechanisms for evoking an emotion by a relevant stimulus have not been clarified. Therefore, there are few reliable techniques for evoking an intended emotion in order to reproduce this process. However, in the field of cognitive science, the ability to alter a bodily response has been shown to unconsciously generate emotions. We therefore hypothesized emotional experience could be manipulated by having people recognize pseudo-generated facial expressions as changes to their own facial expressions. Our results suggest that this system was able to manipulate an emotional state via visual feedback from artificial facial expressions. We proposed the Emotion Evoking system based on the facial feedback hypothesis.
Article
Full-text available
Can technology influence people's mood? This department explores how we might induce emotion through the design of products and user interfaces. The authors describe different ideas and prototypes for motivating people to smile, which they argue will make people happier.
Conference Paper
Computer mediated interaction often lacks of expressivity, in particular for emotion communication. Therefore, we present a concept for context-aware multimodal sharing of emotions for human-to-computer-to-human interaction in social networks. The multimodal inputs and outputs of this system are distributed in a smart environment in order to grant a more immersive and natural interaction experience. The context information is used to improve the opportuneness and the quality of feedback. We implemented an evaluation scenario and we conducted an observation study during some events with the participants. We reported our considerations at the end of this paper.
Article
Full-text available
Two experiments were conducted in which participants looked at photographs (Experiment 1, n = 129) or slides (Experiment 2, n = 90) of people engaging in positive or negative facial expressions. Participants attempted to communicate these facial expressions as accurately as they could to a video camera while viewing themselves in a mirror or without viewing themselves in a mirror. Participants in a control group maintained neutral facial expressions. Participants experienced increased positive moods when they engaged in positive facial expressions and decreased positive moods when they engaged in negative facial expressions. These effects were enhanced when participants viewed themselves in a mirror. The effects of facial expressions on positive affect were stronger for participants with high private self-consciousness. Results were integrated with research identifying individuals who are responsive to self-produced versus situational cues and with theory and research on self-awareness.
Article
Full-text available
Two experiments were conducted in which participants looked at photographs (Experiment 1, n = 129) or slides (Experiment 2, n = 90) of people engaging in positive or negative facial expressions. Participants attempted to communicate these facial expressions as accurately as they could to a video camera while viewing themselves in a mirror or without viewing themselves in a mirror. Participants in a control group maintained neutral facial expressions. Participants experienced increased positive moods when they engaged in positive facial expressions and decreased positive moods when they engaged in negative facial expressions. These effects were enhanced when participants viewed themselves in a mirror. The effects of facial expressions on positive affect were stronger for participants with high private self-consciousness. Results were integrated with research identifying individuals who are responsive to self-produced versus situational cues and with theory and research on self-awareness. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Conference Paper
Full-text available
As the use of mobile data services has spread across the globe, the effect of cultural differences on user requirements has become important issue. To date, however, little research has been conducted on the role cultural factors play in the design of mobile data services. This paper proposes a set of critical design attributes for mobile data services that takes cross-cultural differences into account. To determine these attributes, we devised a qualitative method and conducted in-depth long interviews in Korea, Japan, and Finland. We found 52 attributes considered important by mobile data service users, and 11 critical attributes that showed a clear correlation with characteristics of the user's culture. The paper concludes with a discussion of limitations and of implications for developers of mobile data services.
Conference Paper
Full-text available
This paper proposes a novel interactive technique, the EyeCatcher, which helps photographers capture a variety of natural looking facial expressions of their subjects, by keeping the eyes of the subjects focused on the camera without the stress usually associated with being photographed. We develop a prototype system and verify the effectiveness through evaluation and discussion.
Conference Paper
Full-text available
It can be argued that one of the main reasons for playing games is to achieve an emotional reaction of the player. To be surprised, happy, angry or anxious - to perceive different emotional states - is one of the main reasons to play games. The "Emotional Flowers" game harnesses the player's emotions as the primary means for the game interaction. Within the game the player's facial expression of emotion is used to control the growth of a flower. Multiple players can play "Emotional Flowers" simultaneously. The main idea is to grow the flower as fast as possible based on positive emotions like happiness and surprise. Flowers of all participants within the game are additionally displayed on an ambient display in a public area. This influences not only emotions by the user, but also has an effect on social interactions within the group of players. In this paper, we present design, implementation and evaluation of the "Emotional Flowers" game.
Conference Paper
Full-text available
Perceptual User Interfaces (PUIs) aim at facilitating human-computer interaction with the aid of human-like capacities (computer vision, speech recognition, etc.). In PUIs, the human face is a central element, since it conveys not only identity but also other important information, particularly with respect to the user’s mood or emotional state. This paper describes both a face detector and a smile detector for PUIs. Both are suitable for real-time interaction. The face detector provides eye, mouth and nose locations in frontal or nearly-frontal poses, whereas the smile detector is able to give a smile intensity measure. Experiments confirm that they are competitive with respect to extant detectors. These two detectors are used in an unobtrusive application that allows to interact with an Instant Messaging (IM) client.
Article
Full-text available
Remote measurements of the cardiac pulse can provide comfortable physiological assessment without electrodes. However, attempts so far are non-automated, susceptible to motion artifacts and typically expensive. In this paper, we introduce a new methodology that overcomes these problems. This novel approach can be applied to color video recordings of the human face and is based on automatic face tracking along with blind source separation of the color channels into independent components. Using Bland-Altman and correlation analysis, we compared the cardiac pulse rate extracted from videos recorded by a basic webcam to an FDA-approved finger blood volume pulse (BVP) sensor and achieved high accuracy and correlation even in the presence of movement artifacts. Furthermore, we applied this technique to perform heart rate measurements from three participants simultaneously. This is the first demonstration of a low-cost accurate video-based method for contact-free heart rate measurements that is automated, motion-tolerant and capable of performing concomitant measurements on more than one person at a time.
Article
Full-text available
We investigated the hypothesis that people's facial activity influences their affective responses. Two studies were designed to both eliminate methodological problems of earlier experiments and clarify theoretical ambiguities. This was achieved by having subjects hold a pen in their mouth in ways that either inhibited or facilitated the muscles typically associated with smiling without requiring subjects to pose in a smiling face. Study 1's results demonstrated the effectiveness of the procedure. Subjects reported more intense humor responses when cartoons were presented under facilitating conditions than under inhibiting conditions that precluded labeling of the facial expression in emotion categories. Study 2 served to further validate the methodology and to answer additional theoretical questions. The results replicated Study 1's findings and also showed that facial feedback operates on the affective but not on the cognitive component of the humor response. Finally, the results suggested that both inhibitory and facilitatory mechanisms may have contributed to the observed affective responses.
Conference Paper
With a growing aging population, it has become a very important issue to monitor elderly people who are increasingly living alone and away from their families. Many research projects have explored this issue. However, these are mainly focused on one-way communication. In this paper, we proposed a new communication system for the elderly using a medicine chest. The "Social Medicine Box" is a system which notifies the status of the elderly taking medicines as well as their feeling a picture sent automatically to their family. The elderly can also get feedback and communicate with their family as well as their social network (e.g. Twitter and Facebook). In addition, it allows family members living apart to seamlessly share the information without the annoyance of having to initiate conversation.
Conference Paper
In everyday life, humans interact with many products. In many of these interactions, a person performs an action with, toward, or in the vicinity of a product and then the product reacts to that action. In this paper, however, the opposite interaction pattern, where a product performs an action to induce a user reaction, is presented by a new camera, 'Cheese Cam', concept. Cheese Cam is a camera that can induce unconscious facial reactions in a photography subject, based on mirror neuron theory and facial mimicry theories. A small facial expression icon displayed on Cheese Cam's screen induces unconscious facial reactions in the subject. Experiments were conducted to investigate the effects of Cheese Cam on the facial reactions of subjects, and the results are discussed in this paper. Through this study, we explored possibilities of unconscious interaction.
Conference Paper
Families in Japan increasingly have one or more members living outside of the family household, but many people don't want to lose the bond between family members when they live apart. We have developed a concept called 'Tsunagari' communication aimed at fostering a feeling of connection between people and maintaining their social relationships. A system based on this concept, called the Family Planter system, was also developed for family use. We have field tested this system with family members living apart, and our interviews of users have shown that the users' family relationships tend to be slightly improved by use of this system.
Facial expression analysis, in Handbook of face recognition
  • Y.-L Tian
  • T Kanade
  • J S L Cohn
  • A Jain
Tian, Y.-L., Kanade, T., and Cohn, J. Facial expression analysis, in Handbook of face recognition, S. L. and Jain, A., ed. Springer, October (2003).
America's Families and Living Arrangements
  • U S Bureau
  • U.S. Census Bureau
U.S. Census Bureau. America's Families and Living Arrangements (2007).
Cheese cam: unconscious interaction between humans and a digital camera, CHI '09 Extended Abstracts on Human Factors in Computing Systems
  • Woohun Boram Lee
  • Lee
The negative affects
  • S S Tomkins
Tomkins, S. S. Affect, imagery, consciousness: Vol. 2. The negative affects. Springer, New York, USA, (1963).
The negative affects
  • S S Tomkins
  • Affect
  • Tomkins S. S.
The positive affects
  • S S Tomkins
  • Affect
  • Tomkins S. S.