Content uploaded by Filipa Peleja
Author content
All content in this area was uploaded by Filipa Peleja on Sep 11, 2014
Content may be subject to copyright.
SentiTVchat: Sensing the Mood of Social-TV Viewers
Flávio Martins, Filipa Peleja, João Magalhães
Dep.Informática, Faculdade de Ciências e Tecnologia,
Universidade Nova de Lisboa, Portugal
flaviomartins@acm.org, filipapeleja@gmail.com, jm.magalhaes@fct.unl.pt
ABSTRACT
In this paper, we propose a novel Social-TV chat system
integrating new methods of measuring TV viewers’ feedback and
new multi-screen interaction paradigms. Viewers’ messages in
chat services are a valuable asset for their peers in general, and for
the cable TV operator in particular. The proposed system analyses
chat-messages to detect the mood of viewers towards a given
show (i.e., positive vs negative). This data is plotted on the screen
to inform the viewer about the show popularity. Although the
system provides a one-user / two-screens interaction approach, the
chat privacy is assured by discriminating information sent to the
shared screen or the personal screen. We evaluated the system on
a first experiment with labeled data to assess the accuracy (78%)
of the chat analysis algorithm and a second experiment with live
chat data to validate the user interface.
Categories and Subject Descriptors
H.5.1 [Multimedia Information Systems]
General Terms
Design, Experimentation, Human Factors.
Keywords
SocialTV, chat, sentiment analysis, multi-screen interaction.
1. INTRODUCTION
TV-viewing is still a social experience but nowadays the audience
is connected through the Internet and makes use of social
networks like Twitter and Facebook to share thoughts in real-time
while watching TV content. The living room TV entertainment
environment has been enhanced by new interactive services that
keep the user engaged and active. According to Haythornthwaite
[6] media popularity is linked to social interactions in the new
media. They concluded that social ties and social media are
important to users’ media viewing habits. More recently, Harboe
et al [5], conducted an experiment examining the influence of
Social-TV watching, i.e., users watch television alone but get
instant notifications of what their friends and family are watching.
Oehlberg et al. [8] went one step further and proposed a series of
guidelines for designing distributed Social-TV viewing. In a
related user study Weisz et al. [9], examined the activity of
chatting while watching video online. Their technological solution
was designed to observe human factors and not improve usability.
In contrast, Cattelan et al. [3] implemented a system for chatting
and drawing over video. Zaletelj et al. [10] adapted the live shows
on-the-fly according to users’ explicit votes/audiences collected
from set-top-boxes. Several authors have proposed to analyse user
messages in different domains. Bollen [2] and Dan et al. [4]
addressed the domain of Twitter microposts and proposed to
detect the sentiment expressed in user messages. Recently,
Ariyasu et al. [1] proposed to detect the topics of twitter messages
and to associate them to the correct show. In our system, the show
is known and we proposed to infer the sentiment of the message
towards the show. Thus, we set out to develop a Social-TV system
prototype that allows users to chat with each other, in an
integrated way. The paper’s contribution is twofold:
Multi-screen interaction paradigm: the system User
Interface spans several screens and users interact with media
through two screens. The first screen is intended for media
visualization and it is assumed to be shared. The second
screen is intended for personal use – it is where the user can
watch previews, control the system and chat with peers.
Measuring TV chat feedback: the chat component is
powered by Google Channel API, thus a large base of users
already exist. Sentiment analysis technology observes the
sentiment expressed in opinions towards an item [7]. A novel
viewer feedback metering method is proposed by integrating
sentiment analysis technology into a Social-TV chat system.
With the proposed system, user messages are processed in real-
time by a sentiment analysis algorithm, allowing the plot of the
emotions felt by viewers, Figure 1. Another advantage of the
proposed visualization method is that viewers can get a quick
glimpse of the show’s popularity in the last few minutes.
Figure 1. The SentiTVchat player with the sentiment graph.
In the following section, we shall describe the system architecture
and components, and detail the chat analysis algorithm. An
experimental evaluation and the corresponding discussion
conclude this paper.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
EuroITV’12, July 4–6, 2012, Berlin, Germany.
Copyright 2012 ACM 978-1-4503-1107-6/12/07...$10.00.
161
Figure 2. Media player with SentiTVchat
graph.
Figure 3. Tablet device in
chat mode.
Figure 4. The personal control user
interface.
2. A SYSTEM FOR SOCIAL-TV CHAT
The SentiTVChat system, allows “Social-TV-viewers” to
communicate with each other during TV-viewing activities using
a simple chat interface, shown in Figure 3 and Figure 4. The main
goal was to develop a system for multi-screen TV-viewing
activities and to enable the system to collect chat interactions with
real-time analysis of the chat text to sense user moods.
2.1 System architecture
The system is divided in two parts: a Social-TV service and the
SentiTVchat service, see Figure 5. The Social-TV service is
responsible for the main media application and for binding user
devices in the same session with a common communication
channel. The chat system implements the chat service and
messages analysis technology.
Figure 5. System architecture.
2.2 Social-TV service
Users can access the system by using any Web browser, capable
of running JavaScript, which includes most modern browsers for
desktops, laptops and the browsers included in mobile platforms
such as iOS and Android. In Figure 2, we show the remote-control
client interface running on an Android tablet, for example. The
users can use mobile devices, such as tablets, to remote control the
TV and participate in chat rooms with other people.
TV-Screen: The Media Player UI
To build the TV-screen interface, in Figure 2, we used the
resources provided in the Google TV documentation for HTML5.
In addition, popular browsers started shipping preliminary support
for the Full screen API allowing the player to turn most modern
browsers into a full-blown media consumption screen.
Tablet-screen: The Personal Control UI
The remote-control functionality includes standard controls such
as Play/Pause and other control tasks, it also offers directional
keys for navigating between user-interface buttons on the TV-
screen. After authentication, a bidirectional channel is opened
between the Tablet-screen and the TV-screen. This allows the TV-
screen application to send updated media metadata information
such as program title and progress to the secondary devices.
To authenticate the user, we use two factors: the user’s Google
account login information and a random alphanumeric 5-digit
code that is available to the user in the TV-screen display corner.
This provides better security and allows a single user account to
be able to control a number of TV-screen devices.
Binding devices: a session based communication channel
To bind the user’s devices and to implement communication
channels between them we looked into Web Sockets. Current
browser support for Web Sockets is limited and implementations
differ substantially between browsers providing this API. Thus,
we decided to make use of Google App Engine APIs to build our
system so we could utilize the Channel API available on this
platform. The Channel API is similar to Web Sockets, it allows
Web applications to establish bidirectional communication
channels. However, browser support is improved, since the API is
provided to the client-side in a JavaScript library. This means that
any JavaScript capable mobile browser should be able to
communicate with a multitude of HTML5 ready devices running
the player, such as a Web TV, a PC or a tablet.
2.3 SentiTVchat service
The implemented chat system supports the chat communications
among users seamlessly. The Tablet-side UI gets the current TV-
channel from the updates received from the TV-screen
application. Thus, when users access the chat section in the
Tablet-screen it will automatically enter the corresponding chat
room for the TV-channel being watched on the TV-screen. We
considered that each TV-channel has a chat room with an identical
name, so, if the user is watching the tv1 channel, for example, the
Tablet-screen retrieves the URL: chat-host.tld/?room=tv1. The
page obtained displays the latest 1000 messages from the chat
room. Moreover, a subscription is registered on the server, so that
the client can receive further messages instantly. A subscription is
registered with the chat room, the user, and the communication
channel. When a message is received, it is relayed to all the
subscribers of the corresponding chat room through each of their
channels. However, to send messages, the client makes a POST
request to the /newMessage endpoint instead of using the channel.
162
The system also allows read-only subscriptions to chat rooms,
which do not require authentication. These were used to
implement SentiTVMeter. This feature visualizes the chat room
activity in a novel way: it draws a sliding line chart showing each
message “sentiment analysis” value and a weighted moving
average of the last 20 messages. This visualization was
superimposed into the TV-screen display, so users can always
monitor what is the general sentiment at each moment. The chart
is drawn client-side using the Canvas element and JavaScript,
making it lightweight, because the server just needs to annotate
the messages with the sentiment analysis value, and pipe it
through the communication channel.
2.4 Chat sentiment analysis
Sentiment analysis aims at detecting the preferences expressed
within user-comments. To tackle this problem in an automated
way, comments are represented as a vector of unigrams (words)
which occur within each user-comment. In sentiment analysis not
every word is useful for detecting the preferences of users [7],
thus, in our approach only adjectives, adverbs, verbs and nouns
are considered. In the following sections, the methodology to
represent and classify user-comments will be detailed formally.
2.4.1 Problem formulation
Consider a set of N user-comments, 11
{( , ),...,( , )}
NN
Dcp cp=,
where a user-comment ci is labeled as positive 1
j
p= or
negative 0
j
p=. Taking a machine learning approach and
considering the training set D, sentiment chat analysis aims at
learning a classifier function,
:[0,1]
ii
cpFÎ , (1)
such that for all new user-comment, i
cDÏ, the function F will
infer a polarity value pi = 1 for positive comments and pi = 0 for
negative comments. User-comments, are represented as a vector
of opinion words, i.e., ,1 ,
( ,..., )
ii im
cow ow= where each
component ,im
ow depicts the opinion word m of the user-
comment i. An opinion word is a word that can express a
preference, which can have different intensities and different
orientations (positive or negative).
2.4.2 Orientation and intensity of an OW
The orientation of an OW indicates whether a word expresses a
positive or negative preference. Turney [8] proposed the pointwise
mutual information (PMI) to estimate the orientation of a phrase.
This metric measures the degree of statistical dependence between
two words by observing the probability of co-occurring together
and individually. Hence, in this context the correlation will be
measured by using a positive, and negative, reference. Thereby
one can compute the semantic orientation (SO) of a word as
,
,2
,
(," ")(" ")
()log (," ")(" ")
ij
ij
ij
hits ow excellent hits poor
SO ow hits ow poor hits excellent
æö
÷
ç÷
ç
=÷
ç÷
ç÷
èø
, (2)
where ,
()
ij
hits ow , and for instance ,
(," ")
ij
hits ow excellent , are
given by the number of hits a Web search engine returns using
these keywords. In this paper, we tested the reference words listed
on Table 1.
Table 1. Semantic orientation and word polarity references.
Technique Word polarity references
T: Turney [8] “excellent”/”poor”
G: Generic “good”/”bad”
DS: Domain Specific “best movie”/”worst movie”
DS+T “excellent movie”/”poor movie”
Another factor to be considered is the intensity of an opinion
word. OWs may express different intensity values, such as:
“contented” versus “ecstatic” [7]. Thus, the lexical resource
SentiWordNet [7], defining the intensity of opinion words, is
crucial in this phase. In this lexical resource, each feature is
associated with two numerical scores (positivity and negativity).
In this context, given the SO of an OW, SentiWordNet will return
its sentiment intensity. Therefore, the SentiWordNet (swn) for an
OW will be given by,
(), () 0,
() (), () 0
posSWN ow SO ow
swn ow negSWN ow SO ow
ì
ï>
ï
=í
ï£
ï
î
(3)
where ()posSWN ow corresponds to the positive score value
given by SentiWordNet and ()negSWN ow will correspond to the
negative score. In Table 2 the family column refers to the words’
family: N (noun), V (verb) and R (adverb). Regarding the
sentiment intensity, the lexical resource SentiWordNet associates
each word with a positive (posSWN) and negative (negSWN)
value.
Table 2. SentiWordNet weights for the sentence “Love it or
hate it, however someone tell me what on earth…”.
word family posSWN negSWN
love N 1.375 0.0
hate V 0.0 -0.75
however R 0.5 0.5
someone N 0.0 0.0
tell V 0.875 -0.625
earth N 0.0 -0.625
2.4.3 Classifier function
For classifying reviews, we used a linear classifier to assign a
confidence value to that review. The classifier identifies the
orientation and intensity of all opinion words of a comment
(
)
,1 ,
,...,
iiim
cowow= and computes its rating based on the
sigmoid function,
()
(
)
,
1
1exp
i
ij j
j
cb
ow w
F= +
+å. (4)
The weights
j
w are learned with a gradient descent algorithm to
train the function to rate comments between positive and negative.
The constant b is a bias factor for adjusting the sentiment curve.
2.4.4 Sentiment graph
Since the sentiment graph is plotted on the player screen (which
might be shared), all chat posts are plotted on the graph
anonymously. Figure 1 and Figure 2 shows the sentiment graph on
the player screen: instant messages are plotted in grey and a
163
smoothed curve is plotted in a bright green to offer a better sense
of the show popularity.
3. EXPERIMENTS
The system’s sentiment analysis module was evaluated on two
settings: a first experiment assessed the accuracy of the sentiment
analysis algorithm on labeled data; and a second experiment
validated the system on live chat data captured from Justin.tv.
3.1 Accuracy assessment
The language analyzer Freeling (http://nlp.lsi.upc.edu/freeling)
was applied to identify word families (nouns, verbs, etc). Later,
user-comments were split into sentence level with the Natural
Language Toolkit (http://www.nltk.org/home). A dataset from
IMDB reviews was used to assess the accuracy of the sentiment
analysis algorithm. The data is evenly split in two categories:
positive and negative. We have used 1400 training and 600 test
reviews along with a number of positive and negative reviews
equally divided.
Figure 6: F1-score for IMDB.
Figure 6 shows the classifier performance in terms of F-score, i.e.,
(
)
-score 2 F Precision Recall Precision Recall=+
, also
known as the harmonic mean. This figure illustrates the sentiment
analysis performance under different settings. We can observe
here the importance of choosing adequate reference words for the
Semantic Orientation (Equation 2): the best performance was
achieved with “excellent-movie”/“poor-movie”. This implies that
a combination of a domain specific term (“movie”) with general
string positive and negative terms (“excellent”/”poor”) is the best
setting. These results illustrate the sentiment analysis ability to
evaluate the sentiment within user comments.
3.2 Real-data experiment
To validate the system on real chat data, we collected chat data
from two live Justin.tv channels: 4744 sentences for the IGN Pro
League channel and 4553 sentences for the this WEEK in TECH
channel. Figure 7 illustrates the real-time sentiment analysis graph
of this data. These graphs present the results obtained with an
analysis of the sentiment expressed in the last 100 sentences.
There are several fluctuations on the instant messages (gray line)
and the popularity trends of the show are clearly visible on the red
curve. In this experiment, we observed that the algorithm was not
capable of correctly analyzing some of the messages since viewers
used slang expressions not recognized by the system.
Figure 7. Real-time sentiment analysis graph.
4. DISCUSSION
This paper proposed a novel Social-TV chat system that measures
in real time the viewers’ mood / show popularity. By measuring
and plotting viewers’ mood, users that are zapping through
channels will have instant access to the show popularity over the
last minutes. Additionally, this measure will be closer to real
viewers’ preferences than by simply measuring audiences.
Evaluation on the IMDB dataset demonstrated the ability of the
sentiment classifier in identifying the preferences within user-
comments. As future work, we will improve the sentiment
analysis for other languages and for chat slang.
Acknowledgements. This work has been funded by the
Foundation for Science and Technology by projects UTA-
Est/MAI/0010/2009 and PEst-OE/EEI/UI0527/2011, Centro de
Informática e Tecnologias da Informação (CITI/ FCT/ UNL) -
2011-2012.
5. REFERENCES
[1] Ariyasu, K. et al. 2011. Message analysis algorithms and their
application to social tv. EuroITV ’11, Lisbon, Portugal, Jun. 2011.
[2] Bollen, J. 2010. Determining the public mood state by analysis of
microblogging posts. Alife XII Conf. MIT Press. (2010).
[3] Cattelan, R.G. et al. 2008. Watch-and-comment as a paradigm
toward ubiquitous interactive video editing. ACM Transactions on
Multimedia Computing, Communications, and Applications. (2008).
[4] Dan, O. et al. 2011. Filtering microblogging messages for social tv.
Proceedings of the 20th international conference companion on
World wide web - WWW ’11 (New York, USA, Mar. 2011), 197.
[5] Harboe, G. et al. 2008. Ambient social tv: Drawing people into a
shared experience. ACM SIGCHI Human Factors in Computing
Systems. ACM.
[6] Haythornthwaite, C. 2001. The Strength and the Impact of New
Media. Proceedings of the 34th Annual Hawaii International
Conference on System Sciences ( HICSS-34).
[7] Liu, B. 2010. Sentiment analysis and subjectivity. Handbook of
Natural Language Processing. (2010), 978-1420085921.
[8] Oehlberg, L. et al. 2006. Social TV: Designing for distributed,
sociable television viewing. Proc. EuroITV (2006), 25–26.
[9] Weisz, J.D. et al. 2007. Watching together: Integrating text chat
with video. ACM SIGCHI Human Factors in Computing Systems..
[10] Zaletelj, J. et al. 2009. Real-time viewer feedback in the iTV
production. EuroITV 2009.
0.74 0.74 0.74
0.78
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
T:Turne y G:Ge neric DS: Domain
Specific
DS+T:Domain
Specific+Turne y
F‐score
164