Content uploaded by Faisal Arafsha
Author content
All content in this area was uploaded by Faisal Arafsha on Jul 25, 2022
Content may be subject to copyright.
Design and development of a user centric affective
haptic jacket
Faisal Arafsha &Kazi Masudul Alam &
Abdulmotaleb El Saddik
#Springer Science+Business Media New York 2013
Abstract Affective haptic research is a rapidly growing field. This article intends to improve the
existing literature and contribute by involving consumers directly in the design of a smart haptic
jacket by adding heat, vibration actuators, and by enhancing portability. The proposed system is
designed for six basic emotions: love, joy, surprise, anger, sadness, and fear. Also, it can support
several interacts such as a hug, poke, tickle or touch. An online survey was designed, based on
literature, and conducted on 92 respondents, who gave their opinion about the physiological
impact of emotions and interactions on the human body. The results of this survey assisted in the
general design and implementation of the system. 86 % of the volunteers who participated in the
final experiment expressed their interest in the system and said that the quality of their
multimedia experience was improved through use of the jacket. Detailed design architecture is
provided, along with the details of the hardware and software used for the implementation.
Keywords Wearable haptic jacket .Basic emotion .Vibrotactile actuators .Heat actuators .
Multimedia feedback
1 Introduction
According to Parrot [14], “Emotions are at the heart of social psychology and are part of what
makes social psychology so interesting”. Human emotions can be defined as a set of feelings
that occur in a certain manner and affect behaviour. Some emotions can be evoked socially by
touch, such as a handshake, a hug, or a tickle [9]. Improving emotional involvement, in
entertainment for example, by improving video and audio comes closer to its limits [12]. For
the future of emotions however, research has been concerned for some time about stimulating
Multimed Tools Appl
DOI 10.1007/s11042-013-1767-3
F. Arafsha :K. M. Alam :A. El Saddik (*)
Multimedia Communications Research Lab (MCRLab), School of Electrical Engineering and Computer
Science, University of Ottawa, Ottawa, Ontario, Canada
e-mail: elsaddik@site.uottawa.ca
F. Arafsha
e-mail: faraf069@uottawa.ca
K. M. Alam
e-mail: mkazi078@uottawa.ca
emotions using haptic feedback systems [6,7]. This is also referred to as “Affective Haptics”
[20], which is becoming increasingly popular. It mainly focuses on systems that evoke or affect
ahuman’s emotional condition by the sense of touch. In other words, “Affective haptics”is an
area of research that is being more heavily explored lately. It is concerned with the design of
systems and devices that can intensify, evoke, or further influence the emotional state of a
person by the sense of touch [21].
Involving emotional haptics is also considered an important aspect of research today. There
are some differences in the classifications of basic human emotions amongst researchers and
theorists. However, the majority of them classify basic emotions into “love”,“joy”,“surprise”,
“anger”,“sadness”,and“fear”.Someofthemuse“love”instead of “disgust”[10,13,15,19].
Recent research reveals that the absence of a comforting touch can cause social problems and
mayevenleadtoanearlydeath[8]. In Romania, thousands of orphans lacked affectionate touch
as they were being raised in facilities that were overcrowded, which caused the spread of
emotional and social problems [4]. People want to become more immersed into new technology
and are always looking for something new [12]. The involvement of haptic wearable devices can
also become a major assistance to visually impaired people in the areas of multimedia and social
involvement. Some research is currently involved in designing haptic wearable devices as a
complementary device to audio described movies [22]. In addition, audio/visual descriptions of
multimedia are reaching their limits with today’s advances in technology, such as HD and/or 3D
video and high quality audio. Further improvements are needed for these different aspects [5,12].
This research is intended to incorporate consumers’opinions and currently available haptic
technologies which improve Human Computer Interaction (HCI) by providing consumer-
centric affective haptic clothes. The proposed system will support different applications with
the availability of proper interfacing software design. One application is enhancing video gaming
and movie watching experiences similar to the concept of subtitles in movies. Synchronized
instructions can be sent to the haptic jacket at specific times to evoke emotions based on the
current scene in a movie or a video game. Another possible application for the proposed haptic
jacket is helping visually impaired people understand the emotions of the people in front of them
by analyzing facial expressions. Facial expression recognition for emotional analysis is already
being addressed in current research [24]. Such software can analyze facial expressions and use
the data with simple interfacing software to evoke emotions in the proposed haptic jacket.
2 Related works
Hossain et al. [9] developed interpersonal haptic communication in Second Life
1
where social
physical interactions, such as a handshake, a hug, or a tickle, could stimulate emotions and
cause an emotional reaction for people. This research developed an add-on that works with the
Second Life virtual environment to provide physical communication functions to users
through a haptic jacket. This jacket, using a set of carefully placed vibrotactile actuators,
provides the sensation of touch to the user based on the signals received from the Second Life
add-on. These signals are produced by annotating different parts of the Second Life avatar, and
then actuating the corresponding actuators in the haptic jacket.
Cha et al. [5] implemented HugMe, a synchronous haptic teleconferencing system which
aims to provide the ability to express intimacy remotely. The system presented is compatible
with a tolerable bandwidth of 30–60 Hz for haptic data. Using a 3-DOF force feedback device,
the user can touch a remote user who, in turn, can feel the touch on the contacted skin through
1
Second Life, http://www.secondlife.com
Multimed Tools Appl
a wearable jacket. This jacket contains an array of vibrotactile actuators that activate depending
on the locations of the touches.
A wearable jacket was also developed in Philips Research Europe [11,12] that focuses on
influencing the emotions of movie watchers by creating tactile stimuli to their bodies. The
jacket is body-conforming and contains 64 tactile stimulators. It attempts to increase the user’s
emotional immersion by rendering emotional effects to the user’s skin. This project is based on
the idea that emotions are accompanied by different physical reactions, and that by stimulating
these physical reactions using the jacket, the accompanied emotions could be evoked.
Rahman et al. [1] introduced a custom made web browser with a plug-in that creates tactile
content to be simulated on a wearable vibrotactile jacket. This plug-in enables online video
viewers to feel haptic feedback through the jacket while watching YouTube videos. The plug-
in works by creating a sequence of arrays in XML format. These arrays are composed of
timestamps and actuating values for the operation of the jacket. Viewers can feel the haptic
feedback by choosing YouTube videos that contain annotated files with tactile content, which
is saved on an on-demand server.
Alam et al. introduced complete text analysis for SMS messages looking for secondary
level emotions [14] and communicating them to the users of haptic vibrotactile devices [2].
Using a haptic phone or a haptic jacket, this application prototype maps every emotion into a
distinct vibration pattern. The overall system is composed generally of an emotion server and a
mobile client. The application is in action when the receiving phone forwards received
messages to the remote emotions server (REM) using Simple Object Access Protocol
(SOAP). The message is then analyzed and related emotions are extracted. Finally, emotion
vectors are created and sent back to the phone which, in turn, forwards that vector to the haptic
device (which may be the same phone, another phone, or a haptic jacket).
iFeel_IM! [20,21] introduced a system that uses haptic and visual devices during online
conversations to communicate experienced emotions during that conversation. It consists of
four affective haptic components: HaptiHeart, HaptiHug, HaptiTickler, and HaptiButterfly.
These four affective haptic components activate to represent emotions experienced during
online conversations. Huggy Pajama [17,18] is a wearable and mobile HCI system that allows
its users to remotely feel and hug each other. It is mainly intended for parents who travel a lot
and want to keep in touch with their children and express their affection regularly. The system
is composed of a small, mobile doll that has a pressure sensing circuit embedded in it to sense
different levels of force produced by human interaction (i.e. the hug). Then, it sends the sensed
hug signals to a haptic jacket (or pajama) that, in turn, simulates the measured hug and gives its
wearer the feeling of a hug. The doll (the input device / hugging interface) and the jacket (the
output device) communicate through the internet, and simulate the hug using hardware that
include air pockets, heating elements, and color changing fabric.
The Hug Shirt
2
allows users to send hugs at a distance. The system works by transferring
received hug instructions via SMS to actuators and sensors embedded in the shirt using
Bluetooth technology. Hugs can be sent either by creating a hug SMS instruction using a
provided mobile java application, or by touching specified locations on another input shirt that
is designed for this purpose. The Hug Shirt runs on rechargeable batteries, and is built using
components that comply with the Restriction of use of Hazardous Substances (RoHS
3
).
4
2
“Hug Shirt”, CuteCircuit, http://www.cutecircuit.com/products/thehugshirt/
3
“What is RoHS”, Department for Business Innovation and Skills - National Measurement Office, UK, http://
www.bis.gov.uk/nmo/enforcement/rohs-home
4
“Best Inventions of 2006”, TIME Specials, http://www.time.com/time/specials/packages/article/
0,28804,1939342_1939424_1939709,00.html,2006.
Multimed Tools Appl
From the related studies, it is found that some researchers are focused on the haptic
interfacing aspect, others are focused on the emotional aspect, and a very few are focused
on the integration of emotional stimulation and haptic interfacing in the same system.
Our research contributes to the topic of emotional haptics by designing and
implementing an emotion stimulating haptic jacket which supports six basic emotions
as well as different types of interactions such as a hug, poke, touch, or tickle, and is
based on an empirical study of crowd participation. It includes vibrotactile and heat
actuators in addition to temperature sensors on the body locations agreed upon by the
users.
3 Proposed system and design
In this section, a detailed description of the proposed system is explained including an
explanation of the overall architecture design and the different modules associated with it. It
starts by describing how the data was collected using an online survey and presents the results
of the survey. After that, it explains the general architecture of the implemented system based
on the results of this survey by providing a system overview and an explanation of the
affective haptic components of the system.
3.1 Online survey
The main concept used to construct this survey was based on the list of emotions defined
in the Parrot 2001 [14] model of social psychology as described earlier. The goal was to
figure out which part of the body represents which emotion according to general users. A
total of 92 responses were collected in this survey. Fifty-eight of the respondents were
male and 34 were female. All except for one were between the ages of 18 and 29. The
respondents had varied degree of haptic experiences from none to expert. But, on an
average the population was haptic experienced and most of them are from engineering
background.
3.1.1 Data collection
In the conducted survey, respondents were asked to give their own opinion on how each
(but not necessarily all) of the basic emotions mentioned above can be represented
physically using actuators embedded in smart clothes. In the survey, in addition to
providing the ability for respondents to give their own ideas, three basic types of
feedback were suggested: vibration, beats, and warmth. As a reference, Fig. 1was added
to the survey to assist in providing more accurate and unified results. The areas of
interest are the neck, chest (right and left sides), abdomen (left, middle, right), shoulder
back, back bone, and arms. The Fig. 1was compiled analyzing the available literature
and our intuition.
The survey was created on Survey Monkey
5
as the response collecting link. It was started
on February 8, 2011, and closed on February 22, 2011 (a duration of 2 weeks). The number of
responses varied for each part of the survey because respondents were only asked to answer
the parts they wanted, and had the option to skip questions.
5
www.surveymonkey.com
Multimed Tools Appl
In the survey, we used check boxes to count the number of responses for each type of
feedback, for each body area, and for each emotion. Also, a text box was provided for any
other ideas the users may have (Fig. 1).
3.1.2 Survey results
The results of the conducted survey are described below. Each time a respondent checks a
checkbox in the survey, the count for that specific item is increased. The most interesting and
implementable ideas provided by the respondents are also described below. Overall responses
from the survey participants were analyzed and an overall system was proposed which is
discussed in Section 3.2.
Love (affection, llust, and llonging) The survey results showed that the majority of respon-
dents suggested beats simulation in the jacket to represent “love”, specifically slow beats in the
left chest. Also, it was noticed that, a good number of responses suggested warmth in the neck
and the abdomen area. One written response also suggested warmth all over the body. Total
number of responses: 91.
Joy (cheerfulness, zest, contentment, pride, optimism, enthrallment, and relief) To evoke
“joy”the results of the survey showed that the majority of responses voted for beats in the
left chest side and a slight pressure on the complete chest area. Some of the written responses
suggested the simulation of beats to be soft, and to simulate one long breath. Total number of
responses: 87.
Surprise (amazement, and astonishment) According to the survey, “surprise”can be simulated
by vibration in almost all the upper body in a shiver-like manner, coupled with a sudden,
Fig. 1 Example given to assist in responding to the online survey
Multimed Tools Appl
slightly strong, and fast heartbeat that fades out after a few seconds. Some respondents also
suggested simulating a benign sense of goose bumps, lowering body temperature, and a
hardening of the body. The last part can be implemented by tightening the worn jacket
mechanically. Total number of responses: 88.
Anger (irritation, exasperation, rage, disgust, envy, and torment) “Anger”can be evoked by
some warmth around the neck and vibration in the arms with strong beats on the heart in a
normal speed. Written ideas included tightening the neck and the chest for harder breathing,
and simulating fast breathing. Total number of responses: 90.
Sadness (suffering, disappointment, shame, sympathy, and neglect) According to the survey
results, “sadness”can be emulated by a tightness on the chest using some pressure and
providing some slight vibrations all over the abdomen area. Other written responses included
slow heartbeats and a cold body temperature. Total number of responses: 83.
Fear (horror, and nervousness) Results for “fear”seem to be very similar to “anger”.
However, for fear, vibration is suggested around neck, arm and back bone area and faster
beats on the left chest. Vibration in the backbone will give a shivering experience. Total
number of responses: 89.
At the end of the survey, respondents were asked if having a haptic jacket with a focus on
emotions was of interest to them. The results show that 78 responded positively, 13 responded
negatively, and one respondent skipped this question.
3.2 Affective clothe component organization
Smart clothe, is becoming increasingly popular in the area of human-computer interaction
(HCI). The main focus of affective haptics is on systems that evoke or affect a human’s
emotional condition using the sense of touch [20]. The data collected from the conducted
survey was highly valuable in the overall design of the operation of our system. The designing
decisions were made based on the direct input of the consumers, and the scenarios were agreed
upon as shown in Fig. 2.
As shown in Fig. 2, different scenarios are to be implemented, and different affective haptic
components are required. Based on these scenarios, the system is designed to be composed of
the following affective haptic body parts:
Jacket JðÞ¼fNeck NðÞ;Right Chest RCðÞ;Left Chest LCðÞ;Right Abdomen RAðÞ;
Middle Abdomen MAðÞ;Left Abdomen LAðÞ;Back Shoulder BSðÞ;Back Bone BBðÞ;
Arms AÞgð
In our system, each type of emotional status is represented using the following mechanical
features:
Mechanical Features ¼Warmth WðÞ;Vibration VðÞ;Beats BðÞ;Pressure PðÞ
fg
From Table 1, we find that “love”is represented using warmth and heartbeats. Based on the
users’responses, we have placed warmth and beats on different body parts of the jacket. We
have used vector representation of body parts to formulate different emotional statuses. The
presence of different body parts, such as the neck, abdomen, arms, etc. of the proposed jacket
for a particular emotion, are represented using 1/0 (on/off) in vector J which we call the
localized jacket vector. Additionally, the intensity of one mechanical feature, such as warmth
Multimed Tools Appl
or vibration, on a body part of the jacket is maintained in a range of [0–1]. Each mechanical
feature, which has been used to simulate a basic emotion, is represented by a feature intensity
vector (Table 2) to control its strength on different body parts. Feature intensity vector of
Table 2is produced following empirical studies conducted by haptic and emotion stimulation
domain experts from August 2011–May 2012. Several iterations of jacket component rear-
rangements were made and subsequent user feedbacks were collected to reach a consensus
among domain experts to find average feature intensity value for various mechanical features.
It is to be noted that experts agreed that an ideal solution will be to have an adaptive system
where feature intensities vary according to the user such as male, female, children, visually
impaired etc.
In our system, W
love
is placed in both neck and abdomen parts of the jacket. We maintain
different intensities of warmth in neck and abdomen parts using the feature intensity vector.
Beats simulation is naturally placed in the left side of the chest and it has its own intensity
vector. Hence, the final vector representation of “Love”is W
love
,H
love
.Thejacket’s system
controller generates W
love
and H
love
vectors following Table 2and transmits them to the
microcontroller to simulate the hardware components to produce the right emotional effect.
Fig. 2 Proposed system implementation
Multimed Tools Appl
Here, Love = {W
love
,B
love
}
Wlove ¼Localized Jacket Vector * Feature Intensity Vector
¼Jlove *Ilove
¼1;0;0;1;1;1;0;0;0
fg
*0:4;0;0;0:4;0:4;0:4;0;0;0
fg
¼0:4;0;0;0:4;0:4;0:4;0;0;0
fg
Blove ¼Jlove*Ilove
¼0;0;1;0;0;0;0;0;0
fg
*0;0;0:5;0;0;0;0;0;0
fg
¼0;0;0:5;0;0;0;0;0;0fg
Similarly, Hug is represented using pressure, vibration and beats (Fig. 2). Actuators
following certain pattern are placed in the complete chest and vibration-beat is placed in the
back shoulder. Little vibrationsare alsoused on the middle abdomen.
Hug ¼Phug;Vhug;Bhug
Phug ¼Jhug*I
hug
¼0;1;1;0;0;0;0;0;0
fg
*0;0:7;0:7;0;0;0;0;0;0
fg
¼0;0:7;0:7;0;0;0;0;0;0
fg
Vhug ¼Jhug*I
hug
¼0;0;0;0;1;0;1;0;0
fg
*0;0;0;0;0:4;0;0:5;0;0
fg
¼0;0;0;0;0:4;0;0:5;0;0
fg
Bhug ¼Jhug*I
hug
¼0;0;0;0;0;0;1;0;0
fg
*0;0;0;0;0;0;0:6;0;0
fg
¼0;0;0;0;0;0;0:6;0;0
fg
Tab l e 1 Body local vector for emotions and interactions
NRCLCRAMALABSBBA
Love W 1 0 0 1 1 1 0 0 0
B00100 0000
Joy P 0 1 1 0 0 0 0 0 0
B00100 0000
Surprise V 0 1 1 1 1 1 0 0 1
B00100 0000
Anger W 1 0 0 0 0 0 0 0 0
V00011 1001
B00100 0000
Sadness P 0 1 1 0 0 0 0 0 0
V00011 1000
Fear V 1 0 0 0 0 0 0 1 1
B00100 0000
TickleV00010 1000
Poke B00001 0000
Tou ch V 0 0 0 0 0 0 1 0 0
B00000 0100
P01100 0000
Hug V 0 0 0 0 1 0 1 0 0
B00000 0100
Multimed Tools Appl
Different types of vibrations, pressures, and warmth used in the jacket design are described
below.
&Chest Vibration and Pressure
A number of carefully localized vibrotactile actuators are distributed in the inside of the
jacket in the chest area. They were distributed approximately 12 cm apart in a grid pattern.
These actuators are instructed by the microcontroller to activate in order to simulate the
“surprise”emotion. In order to simulate joy, sadness, or a hug, these same actuators are used
in a different pattern so that a pressure is felt in the chest.
&Neck Vibration
A few small vibrotactile actuators were placed on each side inside of the jacket’s collar.
These actuators vibrate when activating the “fear”emotion in the jacket.
&Arms and Backbone Vibration
A set of vibrotactile actuators were embedded into the inside of the sleeves around the arms.
The microcontroller gives instructions to these actuators to vibrate in order to stimulate “fear”,
“anger”and “surprise”.
A shiver-like feeling is simulated by using the set of vibrotactile actuators inside the
sleeves. The microcontroller is set up to instruct the actuators to vibrate in a specific and
calibrated order for specific durations and give a shiver-like feeling around the arms/back. This
function is used in the case of stimulating “surprise”or “anger”. Five vibrotactile actuators are
placed on the back and front of each bicep, and are programmed to take turns actuating for
Tab l e 2 Mechanical feature intensity vector for emotions and interactions
NRCLCRAMALABSBBA
Love W 0.4 0 0 0.4 0.4 0.4 0 0 0
B000.50 0 0000
Joy P00.40.400 0000
B000.30 0 0000
Surprise V 0 0.4 0.4 0.5 0.5 0.5 0 0 0.3
B000.30 0 0000
AngerW0.3000 0 000 0
V0000.40.40.4000.5
B000.40 0 0000
Sadness P 0 0.7 0.7 0 0 0 0 0 0
V0000.30.30.3000
Fear V 0.3 0 0 0 0 0 0 0.7 0.5
B000.80 0 0000
TickleV0000.70 0.7000
Poke B00000.80000
TouchV00 00 0 00.300
B00000 00.400
Hug P00.70.700 0 000
V 0 0 0 0 0 .4 0 0.5 0 0
B00000 00.600
Multimed Tools Appl
100 ms from shoulder to elbow. The number of vibrators and the durations of vibrations were
selected to represent the shiver feeling in the best way possible after several trials. Similarly,
eight vibrotactile actuators are placed on the backbone and produces the shiver- like experience
for “anger”.
&Abdomen and Shoulder Back Vibration
In the case of abdomen and shoulder back vibration, a number of carefully localized
vibrotactile actuators are distributed on the inside of the jacket. They were distributed approx-
imately 12 cm apart in a grid pattern. These actuators are instructed by the microcontroller to
activate in order to simulate “surprise”,“anger”,“sadness”,“tickle”,“touch”,and“hug”.
&Neck and Abdomen Warmth
Two thermoelectric coolers are also placed on each side inside the jacket’s collar to give a
warm feeling in order to stimulate “love”and “anger”. Also, for love, three thermoelectric
coolers are placed in left, middle and right abdomen. In order to avoid accidental increases in
temperature, tiny electronic thermometers were directly attached to the heat actuators. Their
role is to strictly monitor the temperature of the actuators and give feedback while they are
activated.
&Beat Simulation
It has been proven in research that changing the rate of heartbeats can change a human’s
emotional state, and that each emotion has a different and distinct heartbeat rate [20]. Also, the
Merck Manual of Medical Information states that “the normal heart rate at rest is usually
between 60 and 100 beats per minute”, which is approximately between 1.00 Hz and 1.67 Hz.
However, lower frequencies may also be normal in young adults, especially if they exercise
regularly. Besides responding to exercise and inactivity, a heartbeat’s frequency also responds
to stimuli such as pain and anger [3]. In the implementation of the prototype, this information
was used so that the heartbeat’s rate differed depending on the activated emotion.
3.3 Detailed system architecture
The jacket is designed so that it can be used in a variety of applications. It operates by
connecting to a computer or a pre-set digital system that gives direct instructions and evokes
each emotion for the desired duration.
Here, we used an Arduino Duemilanove
6
microcontroller to control the jacket. The micro-
controller is connected directly to the computer that has an interfacing application. This interfac-
ing application is responsible for giving instructions that the jacket (or the microcontroller) can
understand Fig. 3.
The microcontroller is programmed to receive six simple and direct instructions
representing the six emotions discussed previously (see Fig. 2). Also, it receives the desired
duration associated with each emotion. The format of receivable instructions is a simple one.
The microcontroller receives instructions as a set of three bytes from the interfacing program.
The first byte represents the emotion number (1: love, 2: joy, 3: surprise, 4: anger, 5: sadness,
and 6: fear). The second and third bytes represent the duration for which the emotion shall be
activated in the jacket (in seconds). For example, sending the bytes 456 will activate anger for
56 s, and 654 will activate fear for 54 s.
6
Arduino Duemilanove, http://arduino.cc/en/Main/arduinoBoardDuemilanove
Multimed Tools Appl
The microcontroller’s input and output pins are connected to what we call the “networking
board”. The networking board contains all the circuitry needed to control the jacket. It also
contains necessary voltage regulators to accommodate the different voltage requirements of the
jacket’s components, as well as BJT transistors to control power thirsty components. This
networking board is designed to be connected to the jacket through pluggable wires to make it
easy to improve and to replace parts when necessary.
4 Implementation and evaluation
4.1 Implementation
To implement this system, various hardware and electronic parts are needed. Many products
were dismissed either due to crucial reasons, or simply due to high prices where quality is
almost similar. The following is a description of the main devices used for the implementation
of our system, indicating important characteristics to ensure acceptable quality.
4.1.1 Vibrotactile actuators
Vibrotactile actuators are one of the key requirements for the system since most affective
haptic components highly depend on them. Therefore, the quality of actuators chosen from the
market must be comparatively high. The actuator’s weight should be very light, and barely
noticeable when worn. It must be small in size, must use low voltage (less than 5V), and must
consume very little power. The reason for these requirements is that most microcontrollers
available in the market cannot provide more than 5V, and provide very little current, thus it is
preferred that devices connected to it comply as much as possible.
4.1.2 Heat actuators
For the same reasons mentioned above, heat actuators must be small in size and require no
additional hardware to operate. There exists many heat actuators on the market, but very few
are small in size. The main issue with heat actuators is that they consume a lot of energy in
comparison to any other electronic hardware. One other important issue concerning heat
actuators is the duration they need to warm up and to cool down.
Fig. 3 System architecture
Multimed Tools Appl
4.1.3 Temperature sensors
Temperature sensors are used to ensure safe operation of the heat actuators and to avoid
accidental burns.
4.1.4 Microcontroller
The main points concerning the selection of a suitable microcontroller include the availability
of a sufficient number of controlling I/O pins for the many sensors and actuators connected to
it, providing enough power to operate most of the connected devices, and ease and efficiency
of programming.
4.2 Difficulties in implementation
During the implementation of this system, valuable lessons were learned as some difficulties
arose related to power, wiring, and lack of specific devices. In this section some of the most
important problems are listed, followed by the solutions used to overcome them.
4.2.1 High power requirement
The purchased thermoelectric coolers were the most suitable heat actuators found during this
research. However, they are very high power consumers and require a power supply of around
15V and 7A DC current to operate in ideal conditions. This was an issue since the Arduino
microcontroller board cannot provide sufficient power to meet this high demand. Therefore, an
external power supply device was used in multiple trials with different voltages and current
supplies until the most suitable supply for the system was found at 5.3V and 1.8A supply. The
tests involved trials for connecting the heat devices in parallel and in series, and keeping in
mind that the most acceptable voltage supply was 6V. This was in order to make the system
functional with batteries, as well as to be as portable as possible. In addition, BJT transistors
were used to control the power coming from the external power supply and provide high
current.
4.2.2 Safety
In order to avoid incidental burns from the thermoelectric coolers, it was necessary to monitor
their temperature digitally and control their output. Therefore, digital temperature sensors were
purchased and attached to each one of the thermoelectric coolers.
4.2.3 Calibration
For the shivering sensation on the arms, each vibrotactile actuator had to operate in its own
turn in order to provide a realistic shiver-like feeling. Therefore, each actuator was instructed to
operate for 100 ms, and stop to actuate, then the next one would operate, and so on. Also, to
simulate heartbeats, some intensive research was done until the best timings were agreed upon.
Each heartbeat consists of two separate sub-beat mechanisms, and each sub-beat sounds for a
specific duration. Also, the duration between every sub-beat must be known. This information
was necessary and different for each emotion. Finally, power calibration was needed for the
different affective haptic components in the jacket. Neck vibration, for example, requires very
little DC current, while the ten chest vibrators required much more. The heartbeat vibration
Multimed Tools Appl
actuators also needed to be distinguishable while the chest is vibrating. All these notes were
taken into consideration and different power assignments were given to different parts of the
system.
4.2.4 Synchronization
The Arduino microcontroller cannot be programmed to do multithreading. This was a problem
when actuating heartbeats while another affective haptic component was operating. To over-
come this problem, the code instructions were synchronized so that each affective haptic
component activates at the appropriate instant and for the appropriate duration. In other words,
the overall code was written as if multithreading was predefined.
4.3 Evaluation procedure
The evaluation of the overall system was done in the following steps:
&A media player software was built that plays videos synchronously with timed instructions
to be sent to the haptic jacket. The media player plays six previously selected short videos
(< 1 min each). Each video represents one of the six emotions used in our research. We
also kept four videos (< 30 s) to induce the sensations of touching, tickling, hugging, and
poking.
&Fourteen volunteers who were aged 18–28 years participated in our test. Among them, ten
were males and four were females. They were asked to wear the haptic jacket during the
complete test and each session continued for approximately 45 min with 10 min break. In
the first session, every user has experienced all six videos each on thrice. After each video
watch, they rated the quality of experience (QoE). In the second session, each user
experienced all the actual haptics, interaction videos each one twice, once with haptics
and once without haptics. All the users participated in the tests were greeted with
encouragement remunerations and welcome snacks.
&The first test bed was designed to verify the component organizations of each emotion. In
this case, each emotive video was played once with actual haptic features and twice with
fake haptic features. Fake haptics is a misrepresented haptic pattern according to our
design. For example, when love type of video scenario was played, surprise or anger
haptic pattern was instigated in the jacket. Every user was asked to select all three buttons
of the same row in random order from GUI player. After the completion of one emotion
verification session, users were asked to rate the quality of the experience (QoE) of each.
From this random order, we have classified the QoE results in actual haptics and fake
haptics group (Fig. 5).
&In the second test bed, each video clip was played twice, once keeping the haptic jacket
active with the actual/original haptic features, and once without haptics. The order was
completely random for every video and for every participant. After watching all the videos,
participants were asked to fill a short QoE questionnaire [16]. Touch, poke, tickle and hug
videos were also played in this test bed both with haptic and without haptic features.
4.3.1 Custom built software
The software built specifically for these tests played previously selected videos synchronized
with timed haptic instructions, which were directed towards the haptic jacket. The GUI of this
media player (see Fig. 4) contains a window for viewing the video, buttons for playing
Multimed Tools Appl
emotive/interactive videos, and a checkbox for selecting whether to enable or disable haptic
feedback while playing the video. To provide a non-biased response from the participants, the
buttons contained no description of what type of emotion is represented in the videos. There
are three groups of button actions in the GUI haptic media player where first group represents
actual haptics, second group characterizes fake haptics and third group is for other four
interactions.
4.3.2 The videos
Short videos cannot stimulate emotions as easily as long movies. In a movie, viewers take their
time to understand the idea behind the movie and get involved emotionally. For short videos
however, viewers can make use of the haptic features of the jacket to expedite their emotional
involvement.
4.3.3 Questionnaire
After each participant finished watching the evaluation videos from the second test bed, they
were asked to fill out a short questionnaire composed of eight statements. The first six
statements were selected from a list based on intensive research related to the QoE evaluation
of video games [16]. They were selected based on their relevance to our subject, and to allow a
comparison to the evaluation results of a similar system [12]. The last two statements also
serve our purpose, but were inspired from other research [9]. Each participant was asked to rate
these statements using the 5-point Likert scale, with 5 being Strongly Agree and 1 being
Strongly Disagree. The Likert scale has been commonly used for the cognitive perception
evaluation of users [23]. The statements used in our questionnaire and related results are found
in Table 3.
Fig. 4 GUI of the evaluation media player
Multimed Tools Appl
4.4 Result analysis
In the first test bed (Fig. 5), we verified the component organization of the Haptic clothe. In our
system, every emotion or interaction is represented differently or uniquely in the haptic jacket.
The goal of this test setup was to measure whether the emotion-wise component localization is
recognized, differentiated and accepted by the end user. From Fig. 5we find that our original
orientation of the haptic components (i.e. actual haptics) and corresponding emotion augmen-
tation based movie viewing experience was considerably better than movie viewing with false
or fake haptics for the participants. The users clearly had better immersive experience with our
actual haptics augmentation for all the six emotions.
After the completion of the first test bed, we requested the users to randomly select video
clips from the actual Haptics series i.e. the first column as well as all the interaction (hug,
tickle, touch, and poke) clips one by one. Each video was experienced once with haptics and
once without haptics. These tests were part of the second test bed. After the end of the
complete session, each user was asked to answer the questionnaire of Table 3. The result of
Tab le 3clearly demonstrates that haptics augmented video viewing instils better and
immersive experience over no haptics. In response to the question, whether user would like
Tab l e 3 Users’immersiveness to emotive video with and without haptics
Questionnaire Absent Present
I felt myself “drawn in”2.14 3.94
I enjoyed myself 2.00 4.27
My experience was intense 1.79 3.49
I had a sense of being in the movie scenes 1.43 3.36
I responded emotionally 1.71 3.87
I felt that all my senses were stimulated at the same time 1.64 3.36
The system is easy to get familiar with –4.11
I would consider using the system –3.71
Fig. 5 Average Likert scale rating of emotion wise localized component organization
Multimed Tools Appl
to use our haptic jacket, 86 % answered affirmative. If we compare our results with a similar
Haptic jacket [12] that is intended to influence the emotions of movie watchers by creating
tactile stimuli to their bodies, we find that our system performs better and supports various
emotions as well as different interactions like the hug, poke, touch, and tickle.
5 Conclusion and future work
The research presented in this article outlines a wearable haptic system that is aimed at
enhancing user involvement in movie watching, video gaming, etc., to influence their emo-
tional immersion. The design of this system directly included consumers’opinions on what the
jacket should and should not do. The results of this study assisted greatly in the design of the
basic architecture of the system. The main emotions we focused on were love, joy, surprise,
anger, sadness, and fear. We also included a few other interactions, including poke, tickle,
touch, and hug. We used vibrotactile and heat actuators, along with temperature sensors, in the
implementation of the jacket. 86 % of the respondents expressed interest in our system and are
willing to experience it further. In a nutshell, our proposed system is more engaging and gives
better immersion. For future improvements to this system, air pumping devices may be added
to stimulate pressure and chest tightness in the case of evoking “joy”,“sadness”and “hug”.
Furthermore, since heartbeat rates differ between people depending on their age, sex, fitness,
and as well as other factors, a heartbeat rate sensor can be attached to the jacket system to read
the user’s actual heartbeat rate. The responding stimulation of heartbeats in the jacket system
can be a resulting percentage increase or decrease to the actual rate as read by the sensor,
depending on the intended emotion. In addition, future work for this project can include
analyzing facial expressions and communicating emotions to visually impaired persons to
improve social interaction.
References
1. Abdur Rahman M, Alkhaldi A, Cha J, El Saddik A (2010) “Adding haptic feature to youtube”,MM‘10
proceedings of the international conference on multimedia. ACM, New York
2. Alam KM, El Saddik A, Hardy S, Akther A (2011) SMS text based affective haptic application. Proceedings
of Virtual Reality International Conference (VIRC 2011), April, Laval, France
3. Berkow R, Beers MH, Fletcher AJ (1997) Abnormal heart rhythms. In: The Merck manual of medical
information, home edition. Merck Research Laboratories, New Jersey, p 79
4. Carlson M (1998) Understanding the ‘Mother’s Touch’. Harvard Mahoney Neuroscience Institute Letter to
the Brain 7(1):12–13
5. Cha J, Eid M, Barghout A, Rahman ASMM, El Saddik A (2009) Hugme: synchronous haptic teleconfer-
encing. Proceedings of the seventeenth ACM international conference on Multimedia, pp 1135–1136, New
Yor k , N Y, U S A
6. El Saddik A (2007) The potential of haptics technologies. IEEE Instrum Meas 10:10–17
7. El Saddik A, Orozco M, Eid M, Cha J (2011) Haptics technologies: bringing touch to multimedia. ISBN
978-3-642-22657-1, Springer-Verlag
8. Field T (2002) Infants’need for touch. Hum Dev 45:100–103
9. Hossain SKA, Rahman ASMM, El Saddik A (2010) Interpersonal haptic communication in second life.
IEEE Int Symp Haptic Audio-Visual Environ Games, Oct
10. Izard CE (1977) Human emotions. Plenum Press, New York, p 120
11. Jones WD (2009) Jacket lets you feel the movies. IEEE Spectrum, [online], Available: http://spectrum.ieee.
org/biomedical/devices/jacket-lets-you-feel-the-movies [Dec. 1, 2011]. Mar
Multimed Tools Appl
12. Lemmens P, Crompvoets F, Brokken D, van den Eerenbeemd J, de Vries G (2009) A body-conforming
tactile jacket to enrich movie viewing. EuroHaptics conference, 2009 and Symposium on Haptic Interfaces
for Virtual Environment and Teleoperator Systems, pp 7–12
13. Ortony A, Turner TJ (1990) What’s basic about basic emotions? Psychol Rev 97:315–331
14. Parrott WG (2001) Emotions in social psychology. Psychology Press, Philadelphia
15. Plutchik R (1980) A general psychoevolutionary theory of emotion. In Plutchik R, Kellerman H (eds)
emotion: theory, research, and experience: theories of emotion. New York, vol. 1, pp. 3–33
16. Rajae-Joordens RJE (2008) Measuring experiences in gaming and tv applications—investigating
the added value of a multi-view auto-stereoscopic 3D display. In Westerink JWD, et al. (ed)
Probing experience—from assessment of user emotions and behaviour to development of products,
pp. 77–90
17. Teh JKS, Cheok AD, Choi Y, Fernando CL, Peiris RL, Fernando ONN (2009) Huggy Pajama- a parent and
child hugging communication system. Proceedings of the 8th International Conference on Interaction Design
and Children IDC, pp. 290–291, Como, Italy
18. Teh JKS, Cheok AD, Peiris RL, Choi Y, Thuong V, Lai S (2008) Huggy Pajama—a mobile parent and child
hugging communication system. Proceedings of the 7th International Conference on Interaction Design and
Children IDC, pp. 250–257, Chicago, IL, USA
19. Tomkins SS (1984) Affect theory. In: Scherer KR, Ekman P (eds) Approaches to emotion. Lawrence
Erlbaum, Hillsdale, pp 163–196
20. Tsetserukou D, Neviarouskaya A (2010) iFeel_IM!: augmenting emotions during online communication.
IEEE Comput Graph Appl 30(5):72–80
21. Tsetserukou D, Neviarouskaya A, Prendinger H, Kawakami N, Tachi S (2009) Affective haptics in emotional
communication. 3rd International Conference on Affective Computing and Intelligent Interaction and
Workshops, ACII, pp. 1–6
22. Viswanathan LN, McDaniel T, Krishna S, Panchanathan S (2010) Haptics in audio described
movies. 2010 I.E. International Symposium on Haptic Audio-Visual Environments and Games
(HAVE), pp. 1–2
23. Wu W, Arefin A, Rivas R, Nahrstedt K, Sheppard R, Yang Z (2009) Quality of experience in distributed
interactive multimedia environments: toward a theoretical framework. Proceedings of the 17th ACM
International Conference on Multimedia MM ‘09,NewYork,NY,USA
24. Yun T, Guan L (2009) Automatic fiducial points detection for facial expressions using scale invariant feature.
IEEE International Workshop on Multimedia Signal Processing MMSP ‘09, October
Faisal Arafsha received his M.A.Sc. degree in Electrical and Computer Engineering from the University of
Ottawa in Canada early 2012, and his B.Sc. in Computer Engineering from King Fahd University of Petroleum
and Minerals (KFUPM) in Saudi Arabia in 2009. He is now a Ph.D. candidate at the University of Ottawa and
has been working in the Embedded Computing Systems and Haptic Feedback research in the MCRLab since
2010. Before that, Faisal has worked in the automation industry in one of the largest oil and gas corporations
worldwide.
Multimed Tools Appl
Kazi Masudul Alam received his M.A.Sc degree in Computer Science from University of Ottawa, Canada in
2012, and his B.Sc in Computer Science and Engineering from Khulna University, Bangladesh in 2007. He is
now a Ph.D. candidate at the University of Ottawa and has been working on Human Computer Interaction and
3D BigData Visualization. His master’s thesis was related to design and development of next generation E-Book
reader system to provide immersive reading experience. He has authored and co-authored 15 peer reviewed
research articles in renowned conferences and journals. He is a faculty member (on leave) of Khulna University
Computer Science and Engineering discipline and also works as independent Information Technology specialist.
Abdulmotaleb El Saddik (F’2009) is University Research Chair and Professor in the School of Electrical
Engineering and Computer Science at the University of Ottawa. He held regular and visiting positions in Canada,
Spain, Saudi Arabia, UAE, Germany and China. He is an internationally-recognized scholar who has made strong
contributions to the knowledge and understanding of multimedia computing, communications and applications.
He has authored and co-authored four books and more than 400 publications. Chaired more than 40 conferences
and workshop and has received research grants and contracts totaling more than /18 Mio. He has supervised more
than 100 researchers. He received several international awards among others ACM Distinguished Scientist,
Fellow of the Engineering Institute of Canada, Fellow of the C anadian Academy of Engineers and Fellow of IEEE
and IEEE Canada Computer Medal.
Multimed Tools Appl
