Conference PaperPDF Available

Live Sentiment Annotation of Movies via Arduino and a Slider


Abstract and Figures

In this contribution, we present the first version of a novel approach and prototype to perform live sentiment annotation of movies while watching them. Our prototype consists of an Arduino microcontroller and a potentiometer, which is paired with a slider. We motivate the need for this approach by arguing that the presentation of multimedia content of movies as well as performing the annotation live during the viewing of the movie is beneficial for the annotation process and more intuitive for the viewer/annotator. After outlining the motivation and the technical setup of our system, we report on which studies we plan to validate the benefits of our system.
Content may be subject to copyright.
Live Sentiment Annotation of Movies via Arduino and a Slider
Thomas Schmidt & David Halbhuber
Media Informatics Group, University of Regensburg, Regensburg, Germany
Abstract. In this contribution, we present the first version of a novel approach
and prototype to perform live sentiment annotation of movies while watching
them. Our prototype consists of an Arduino microcontroller and a potentiome-
ter, which is paired with a slider. We motivate the need for this approach by ar-
guing that the presentation of multimedia content of movies as well as perform-
ing the annotation live during the viewing of the movie is beneficial for the an-
notation process and more intuitive for the viewer/annotator. After outlining the
motivation and the technical setup of our system, we report on which studies we
plan to validate the benefits of our system.
Keywords: Sentiment Annotation, Sentiment Analysis, Movies, Movie Anno-
tation, Arduino, Slider
1 Motivation
The research areas concerned with the computational prediction and analysis of sen-
timents and emotions, predominantly in written text, are referred to as sentiment and
emotion analysis respectively [16]. The main application area of sentiment analysis is
user generated content on the web like social media [8] or movie reviews [12]. How-
ever, in recent years, sentiment and emotion analysis has been explored in the context
of the Digital Humanities (DH), more precisely Computational Literary Studies.
1.1 Sentiment and Emotion Analysis in Digital Humanities
Overall, one can identify an increasing interest for sentiment analysis among literary
studies in recent years. Researchers explore the results of the application of sentiment
and emotion analysis in various literary genres like plays [18, 20, 24], novels [11] and
fairy tales [1, 2, 18]. Sentiment analysis is also used to predict “happy endings” in
novels [9], to identify plot arcs in stories [23], to predict genres [15] or to compare the
sentiment expression in the text and the audiobook of a play [26]. A more in-depth
analysis of the state of sentiment analysis research in Digital Humanities can be found
in Kim and Klinger [13].
Currently, the focus of research is predominantly on text, esp. traditional text gen-
res like novels and plays. While there are some first explorations of multimedia con-
tent like audiobooks [26], the application of sentiment analysis on movies has hardly
been explored so far: Öhman and Kajava [21] have developed Sentimentator, an an-
Please cite as:
Schmidt, T. & Halbhuber, D. (2020). Live Sentiment Annotation of Movies via
Arduino and a Slider. In Digital Humanities in the Nordic Countries 5th Confer-
ence 2020 (DHN 2020). Late Breaking Poster. Riga, Latvia.
notation tool specifically designed to annotate sentiment and emotion for movie subti-
tles. Chu and Roy [6] explore multimodal sentiment analysis in videos and focus on
short web videos to identify emotional arcs.
We argue that similar to literary studies, emotions and sentiments of the characters
in the movie play an important role in the analysis and interpretation of a movie. Ad-
vances in sentiment analysis for this type of media could lead to large-scale analysis
examining sentiment and emotion progressions and distribution in entire genres or
epochs. Insights gained by such analysis are of large interest for film scholars. For
such large-scale movie analysis via computational methods, the term “Distant View-
ing” was framed [1, 4].
1.2 Sentiment and Emotion Annotation of Literary Texts and Movies
An important part in the process of sentiment analysis for a specific domain is the
creation of large, well-curated corpora annotated with sentiment and emotion infor-
mation. These corpora allow for the application of advanced machine learning ap-
proaches but also for systematic performance evaluations of various approaches.
While these corpora are rather easy to acquire in the context of social media and
product reviews e.g. by using Amazon Mechanical Turk [29] it has been shown that
the annotation of literary texts is more challenging [2, 25, 26]. Due to the current lack
of annotated corpora, researchers often employ rule-based sentiment analysis ap-
proaches (dictionary-based approaches; c.f. [13]). These are oftentimes not optimized
for the specific domain and vocabulary of literary texts and are often out-performed
by machine learning approaches [13, 29].
There are various reasons for the lack of annotated corpora for literary texts. Sen-
timent annotation for this text sort is currently done with text-based annotation tools
enabling the annotation of passages [25, 26] or more complex relations in the text
[14]. However, annotators perceive the task as very challenging and tedious [2, 25]. If
the annotators have no specific expertise, they report many problems with the lan-
guage and the missing context [2, 25, 27]. Furthermore, narrative texts are generally
more prone to subjectivity since they can be interpreted in different ways. Therefore,
annotation agreements are rather low [2, 25, 27], which is also a problem for the suc-
cessful creation of corpora and the application of machine learning. In the context of
movies, sentiment or emotion annotation projects are rare and mostly focused on the
annotation of the textual content of movies like the subtitles [21]. Due to similar chal-
lenges like the ones with literary texts, Öhman and Kajava [21] developed a tool em-
ploying gamification to facilitate the sentiment and emotion annotation process for
movie subtitles.
1.3 The Need for Live Sentiment Annotation
In the following, we present an annotation process using an Arduino and sensor slid-
ers for live sentiment annotation of movies. Annotators can perform the annotation
while watching a movie on a TV screen and adjust the sentiment during the viewing
of the movie via a continuous slider (e.g. from negative to positive sentiment) at the
side of their chair (see chapter 2). The changes of the slider are saved via a Python
script and can be aligned with the movie.
We argue that the lack of the presentation of the audiovisual modalities , which are
of course important parts of movies, and the sole focus on text in sentiment annotation
for movies can lead to many of the problems concerning sentiment and emotion anno-
tation mentioned in chapter 1.2. We want to highlight that textual sentiment annota-
tion of movies might still be reasonable for certain tasks e.g. multilingual analysis of
emotions of subtitles [22]. However, for our specific goal of the annotation of the
expressed sentiment of the characters in a movie, we argue that our approach has
multiple benefits.
First, movies are multimedia artefacts and the lack of the presentation of the video
and audio channels leads to information loss. Many emotions are expressed via the
face and the voice of the actor and not just the text. Therefore, viewers might be able
to annotate sentiment and emotions easier and more consistent when experiencing the
entire movie, thus leading to higher agreement levels among annotators. Additionally,
a lot of context that might be important to understand the feelings of the characters
might be expressed via other channels than the text. Furthermore, emotions are also
often expressed without saying anything in a movie. Textual annotation only allows
the annotation of parts in which characters talk, everything else is neglected. This will
certainly lead to incorrect and incomplete sentiment annotations for certain movies.
While there are video annotation tools that offer the video and audio channel to be
used for movie annotation, they often need training before usage and rather support
asynchronous work needing to constantly pause and adjust the time and frame of the
movie for the annotation (e.g. [7, 17]). We, however, argue that live annotation during
the viewing of the movie facilitates the annotation process because the view-
er/annotator can directly and immediately assign their annotations based on what they
are experiencing. Furthermore, the usage of a continuous slider might also resemble
the rather vague concept of sentiment much more than nominal class assignments [5,
24] or ordinal ratings [19, 28] often used in textual sentiment annotation. Following
Nobel laureate Daniel Kahneman’s line of thought, annotating in the actual movie
watching situation might come closer to emotional reality than more reflective post-
hoc annotation [10]. Finally, we argue that our annotation process is more in line with
the private viewing behavior and annotators do not need to learn and use complex
annotation tools; thus the annotation might not be perceived as much as tedious work
than it has been in the context of textual sentiment annotation of literary texts.
Please note that while we focus on sentiment annotation for our study, the system
can be similarly used for every sort of emotion or other scale for which one desires
live movie annotations. Furthermore, our motivation for live multimedia annotation
also holds true for theatre plays (and performances) which, similar to movies, are
mostly analyzed based on their textual content [18, 20, 24]. Our system is not depend-
ent of the content of the presented video and can easily be applied to theater record-
ings but also TV series or other kinds of videos. Additionally, while the task in our
annotations is to mark the sentiment of the characters on the screen, our system can
also be used in viewer response studies of various forms in psychology.
2 Approach
In this chapter, we sketch our technical setup and describe the actual annotation pro-
2.1 Technical Setup
The annotation system consists of an Arduino microcontroller connected to a linear
potentiometer, which, is paired with a slider. Figure 1 shows the system’s hardware.
Fig.1. Arduino (blue) connected to a slider with integrated potentiometer (red).
The Arduino itself has to be connected to a computer running a Python script. The
script represents the core of the system, it is responsible for reading the current value
of the slider, logging it and presenting it to the user in a small GUI while watching the
movie on a TV. The slider depicts continuously changing resistance levels between 0
and 1023; these values may be translated programmatically to other scales. The py-
thon script, running in the background (e.g. on a laptop connected to the TV), records
these values simultaneously and it shows the user the current slider position and thus
the currently selected value in a simple GUI. Figure 2 depicts the user view for an
exemplary application in a movie annotation.
Fig. 2. Example scene from a TV show (left). Python script displaying the currently chosen
value on the Arduino slider. The GUI also depicts a rudimentary scale (right).
2.2 Annotation Process
For the annotation process, the annotator/viewer will be presented with the movie and
the interface as shown in figure 2. Additionally, the annotators are equipped with the
cased Arduino slider. Figure 3 shows an early prototype of the sliders encasing.
Fig. 3. Prototype of the slider casing. The shell also features a rudimentary scale allowing the
user to navigate without the GUI. Please note that the slider is continuous and not nominal.
The slider will be operated by the user while watching a movie or TV show and is
placed at the side of his chair so the viewer can adjust the scale intuitively with his
hands, while watching the movie (figure 4). The slider is portable and can be placed
as the viewer wishes. During the study, the value of the Arduino slider, time-stamped,
is read and logged by the Python script every 100ms. By saving the timestamp, the
slider value can be exactly assigned to a certain time in a film or TV program in a
subsequent data analysis. The movie shown as well as the slider are synchronized via
a Python script connecting the slider and VLC-player, which is the media player we
use to present the movie.
To start the annotation, we simply connect a laptop with a TV and start the script
and a movie via VLC-player. The annotators can also stop and continue the movie as
they wish without deranging the synchronization. The final output is currently a sim-
ple table with the value of the slider for every 100ms. Via post processing, we can
therefore precisely acquire and visualize the sentiment progression of a movie accord-
ing to the annotator.
Fig. 4. Slider case at the side of the seat during a live annotation.
3 Future Plans
While we argue that our concept is beneficial for the sentiment annotation process of
movies, we have yet to evaluate our approach to validate the assumptions we make in
chapter 1. We currently plan first sentiment annotation studies for movies to evaluate
the system. To perform this evaluation, an adequate number of annotators will per-
form the annotation of a small number of movies via our system. They will be intro-
duced to the annotation process and then have to watch a movie and annotate the sen-
timent expressed by the characters in the film. For valid evaluations, we plan to com-
pare this annotation process with (1) the same type of annotations of solely the textual
content of the movies and general annotation tools as well as with (2) video annota-
tion tools. We will analyze the differences in annotation behavior as well as the per-
ceived effort via standard usability metrics. We will also examine how computational
sentiment analysis approaches, being textual or multimodal, perform in light of the
annotation and which approach mostly resembles the annotations by viewers.
To gather valid and large-scale corpora with sentiment or emotion annotations, it is
important to find a fitting and easy-to-perform annotation process. While the ap-
proach is still limited since the annotation of one person takes at least the time of one
movie, we argue that it is a step forward compared to traditional annotation methods.
Furthermore, even with a moderate size of annotated movies we can work towards the
usage of advanced machine learning approaches.
1. Alm, C.O., Roth, D., Sproat, R.: Emotions from text: machine learning for textbased emo-
tion prediction. In: Proceedings of the conference on human language technology and em-
pirical methods in natural language processing. pp. 579586. Association for Computa-
tional Linguistics (2005)
2. Alm, C.O., Sproat, R.: Emotional sequencing and development in fairy tales. In: Interna-
tional Conference on Affective Computing and Intelligent Interaction. pp. 668674.
Springer (2005)
3. Arnold, T., Tilton, L.: Distant viewing: analyzing large visual corpora. Digital Scholarship
in the Humanities 34(Supplement 1), i3i16 (2019)
4. Bender, K.: Distant viewing in art history. a case study of artistic productivity. Internation-
al Journal for Digital Art History (1) (2015)
5. Bosco, C., Allisio, L., Mussa, V., Patti, V., Ruffo, G.F., Sanguinetti, M., Sulis, E.: Detect-
ing happiness in italian tweets: Towards an evaluation dataset for sentiment analysis in fe-
licitta. In: 5th International Workshop on EMOTION, SOCIAL SIGNALS, SENTIMENT
& LINKED OPEN DATA, ES3LOD 2014. pp. 5663. European Language Resources As-
sociation (2014)
6. Chu, E., Roy, D.: Audio-visual sentiment analysis for learning emotional arcs in movies.
In: 2017 IEEE International Conference on Data Mining (ICDM). pp. 829 834. IEEE
7. Dutta, A., Zisserman, A.: The via annotation software for images, audio and video. In:
Proceedings of the 27th ACM International Conference on Multimedia. pp. 22762279
8. Hutto, C.J., Gilbert, E.: Vader: A parsimonious rule-based model for sentiment analysis of
social media text. In: Eighth international AAAI conference on weblogs and social media
9. Jannidis, F., Reger, I., Zehe, A., Becker, M., Hettinger, L., Hotho, A.: Analyzing features
for the detection of happy endings in german novels. arXiv preprint arXiv:1611.09028
10. Kahneman, D.: Thinking, fast and slow. Macmillan (2011)
11. Kakkonen, T., Kakkonen, G.G.: Sentiprofiler: Creating comparable visual profiles of sen-
timental content in texts. In: Proceedings of the Workshop on Language Technologies for
Digital Humanities and Cultural Heritage. pp. 6269 (2011)
12. Kennedy, A., Inkpen, D.: Sentiment classification of movie reviews using contextual va-
lence shifters. Computational intelligence 22(2), 110125 (2006)
13. Kim, E., Klinger, R.: A survey on sentiment and emotion analysis for computational liter-
ary studies. arXiv preprint arXiv:1808.03137 (2018)
14. Kim, E., Klinger, R.: Who feels what and why? annotation of a literature corpus with se-
mantic roles of emotions. In: Proceedings of the 27th International Conference on Compu-
tational Linguistics. pp. 13451359 (2018)
15. Kim, E., Padó, S., Klinger, R.: Prototypical emotion developments in literary genres. In:
Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural
Heritage, Social Sciences, Humanities and Literature. pp. 1726 (2017)
16. Liu, B.: Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge Uni-
versity Press (2016)
17. Martin, J.C., Kipp, M.: Annotating and measuring multimodal behaviour-tycoon metrics in
the anvil tool. In: LREC. Citeseer (2002)
18. Mohammad, S.: From once upon a time to happily ever after: Tracking emotions in novels
and fairy tales. In: Proceedings of the 5th ACL-HLT Workshop on Language Technology
for Cultural Heritage, Social Sciences, and Humanities. pp. 105114. Association for
Computational Linguistics (2011)
19. Momtazi, S.: Fine-grained german sentiment analysis on social media. In: LREC. pp.
12151220. Citeseer (2012)
20. Nalisnick, E.T., Baird, H.S.: Character-to-character sentiment analysis in shakespeare's
plays. In: Proceedings of the 51st Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers). vol. 2, pp. 479483 (2013)
21. Öhman, E., Kajava, K.: Sentimentator: Gamifying fine-grained sentiment annotation. In:
DHN. pp. 98110 (2018)
22. Öhman, E., Kajava, K., Tiedemann, J., Honkela, T.: Creating a dataset for multilingual fi-
ne-grained emotion-detection using gamification-based annotation. In: Proceedings of the
9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media
Analysis. pp. 2430 (2018)
23. Reagan, A.J., Mitchell, L., Kiley, D., Danforth, C.M., Dodds, P.S.: The emotional arcs of
stories are dominated by six basic shapes. EPJ Data Science 5(1), 31 (2016)
24. Schmidt, T., Burghardt, M.: An evaluation of lexicon-based sentiment analysis techniques
for the plays of Gotthold Ephraim Lessing. In: Proceedings of the Second Joint SIGHUM
Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humani-
ties and Literature. pp. 139149. Association for Computational Linguistics (2018),
25. Schmidt, T., Burghardt, M., Dennerlein, K.: Sentiment annotation of historic german
plays: An empirical study on annotation behavior. In: Kübler, S., Zinsmeister, H. (eds.)
Proceedings of the Workshop for Annotation in Digital Humantities (annDH). pp. 47–52.
Sofia, Bulgaria (August 2018), 2155/schmidt.pdf
26. Schmidt, T., Burghardt, M., Wolff, C.: Toward multimodal sentiment analysis of historic
plays: A case study with text and audio for Lessing's Emilia Galotti. In: Proceedings of the
Digital Humanities in the Nordic Countries Conference 2019 (DHN 2019). pp. 405414
27. Sprugnoli, R., Tonelli, S., Marchetti, A., Moretti, G.: Towards sentiment analysis for his-
torical texts. Digital Scholarship in the Humanities 31(4), 762772 (2016)
28. Takala, P., Malo, P., Sinha, A., Ahlgren, O.: Gold-standard for topic-specific sentiment
analysis of economic texts. In: LREC. vol. 2014, pp. 21522157 (2014)
29. Vinodhini, G., Chandrasekaran, R.: Sentiment analysis and opinion mining: a survey. In-
ternational Journal 2(6), 282292 (2012)
... This is regarded as moderate agreement (Landis and Koch, 1977), which is low compared with sentiment analysis research with other text sorts (cf. Mäntylä et al., 2018) but in line with similar annotation projects with literary and historical texts (Alm and Sproat, 2005;Sprugnoli et al., 2015;Schmidt et al., , 2019aÖhman, 2020;Schmidt et al., 2020a). ...
Full-text available
Conference Paper
We present results of a project on emotion classification on historical German plays of Enlightenment , Storm and Stress, and German Classicism. We have developed a hierarchical annotation scheme consisting of 13 sub-emotions like suffering, love and joy that sum up to 6 main and 2 polarity classes (positive/negative). We have conducted textual annotations on 11 German plays and have acquired over 13,000 emotion annotations by two annotators per play. We have evaluated multiple traditional machine learning approaches as well as transformer-based models pretrained on historical and contemporary language for a single-label text sequence emotion classification for the different emotion categories. The evaluation is carried out on three different instances of the corpus: (1) taking all annotations, (2) filtering overlapping annotations by annotators, (3) applying a heuristic for speech-based analysis. Best results are achieved on the filtered corpus with the best models being large transformer-based models pretrained on contemporary German language. For the polarity classification accuracies of up to 90% are achieved. The accuracies become lower for settings with a higher number of classes, achieving 66% for 13 sub-emotions. Further pretraining of a historical model with a corpus of dramatic texts led to no improvements .
... The subjectivity in the interpretation of literary material is not affected by the presentation of the video channel. We want to pursue improvements for the annotation process by developing specific video annotation tools, enabling the annotation while watching the movie [42,43] While the presented case study can be regarded as negative result we did learn that the application of general-purpose sentiment analysis is not sufficient for our material. Thus, we are currently conducting larger annotation studies to gather training material for optimized machine learning approaches but also to explore the influence of multimodality on the annotation procedure. ...
Full-text available
Conference Paper
We present first results of an exploratory study about sentiment analysis via different media channels on a German historical play. We propose the exploration of other media channels than text for sentiment analysis on plays since the auditory and visual channel might offer important cues for sentiment analysis. We perform a case study and investigate how textual, auditory (voice-based), and visual (face-based) sentiment analysis perform compared to human annotations and how these approaches differ from each other. As use case we chose Emilia Galotti by the famous German playwright Gotthold Ephraim Lessing. We acquired a video recording of a 2002 theater performance of the play at the "Wiener Burgtheater". We evaluate textual lexicon-based sentiment analysis and two state-of-the-art audio and video sentiment analysis tools. As gold standard we use speech-based annotations of three expert annotators. We found that the audio and video sentiment analysis do not perform better than the textual sentiment analysis and that the presentation of the video channel did not improve annotation statistics. We discuss the reasons for this negative result and limitations of the approaches. We also outline how we plan to further investigate the possibilities of multimodal sentiment analysis.
... Quantitative methods have a long tradition in film analysis going back to the predigital era (Salt 1974;Vonderau 2020). Nowadays, multiple projects explore movies via computational methods to investigate colors (Burghardt et al. 2016;Flueckiger 2017;Kurzhals et al. 2016;Masson et al. 2020;Pause / Walkowski 2018), shot lengths (Baxter et al. 2017;DeLong 2015) or annotation possibilities (Halter et al. 2019;Kuhn et al. 2015;Schmidt / Halbhuber 2020;Schmidt et al. 2020a). Recent research has also led to the definition of the term Distant Viewing (Arnold / Tilton 2019) to describe large-scale digital movie analysis. ...
Full-text available
Conference Paper
We present an exploratory study in the context of digital film analysis inspecting and comparing five canonical movies by applying methods of computer vision. We extract one frame per second of each movie which we regard as our sample. As computer vision methods we explore image-based object detection, emotion recognition, gender and age detection with state-of-the-art models. We were able to identify significant differences between the movies for all methods. We present our results and discuss the limitations and benefits of each method. We close by formulating future research questions we plan to answer by applying and optimizing the methods.
Full-text available
Conference Paper
We introduce Sentimentator; a publicly available gamified web-based annotation platform for fine-grained sentiment annotation at the sentence-level. Sentimentator is unique in that it moves beyond binary classification. We use a ten-dimensional model which allows for the annotation of 51 unique sentiments and emotions. The platform is gamified with a complex scoring system designed to reward users for high-quality annotations. Sentimentator introduces several unique features that have previously not been available, or at best very limited, for sentiment annotation. In particular, it provides streamlined multi-dimensional annotation optimized for sentence-level annotation of movie subtitles. Because the platform is publicly available it will benefit anyone and everyone interested in fine-grained sentiment analysis and emotion detection, as well as annotation of other datasets.
Full-text available
Conference Paper
We present results of a sentiment annotation study in the context of historical German plays. Our annotation corpus consists of 200 representative speeches from the German playwright Gotthold Ephraim Lessing. Six annotators, five non-experts and one expert in the domain, annotated the speeches according to different sentiment annotation schemes. They had to annotate the differentiated polarity (very negative, negative, neutral, mixed, positive, very positive), the binary polarity (positive/negative) and the occurrence of eight basic emotions. After the annotation, the participants completed a questionnaire about their experience of the annotation process; additional feedback was gathered in a closing interview. Analysis of the annotations shows that the agreement among annotators ranges from low to mediocre. The non-expert annotators perceive the task as very challenging and report different problems in understanding the language and the context. Although fewer problems occur for the expert annotator, we cannot find any differences in the agreement levels among non-experts and between the expert and the non-experts. At the end of the paper, we discuss the implications of this study and future research plans for this area.
Full-text available
Conference Paper
We present results from a project on sentiment analysis of drama texts, more concretely the plays of Gotthold Ephraim Lessing. We conducted an annotation study to create a gold standard for a systematic evaluation. The gold standard consists of 200 speeches of Lessing's plays and was manually annotated with sentiment information by five annotators. We use the gold standard data to evaluate the performance of different German sentiment lexicons and processing configurations like lemmatization, the extension of lexicons with historical linguistic variants, and stop words elimination, to explore the influence of these parameters and to find best practices for our domain of application. The best performing configuration accomplishes an accuracy of 70%. We discuss the problems and challenges for sentiment analysis in this area and describe our next steps toward further research.
Full-text available
Conference Paper
We present a case study as part of a work-in-progress project about multimodal sentiment analysis on historic German plays, taking Emilia Galotti by G. E. Lessing as our initial use case. We analyze the textual version and an audio version (audiobook). We focus on ready-to-use sentiment analysis methods: For the textual component, we implement a naive lexicon-based approach and another approach that enhances the lexicon by means of several NLP methods. For the audio analysis, we use the free version of the Vokaturi tool. We compare the results of all approaches and evaluate them against the annotations of a human expert, which serves as a gold standard. For our use case, we can show that audio and text sentiment analysis behave very differently: textual sentiment analysis tends to predict sentiment as rather negative and audio sentiment as rather positive. Compared to the gold standard, the textual sentiment analysis achieves accuracies of 56% while the accuracy for audio sentiment analysis is only 32%. We discuss possible reasons for these mediocre results and give an outlook on further steps we want to pursue in the context of multimodal sentiment analysis on historic plays.
Full-text available
Conference Paper
Most approaches to emotion analysis in fictional texts focus on detecting the emotion expressed in text. We argue that this is a simplification which leads to an overgeneralized interpretation of the results, as it does not take into account who experiences an emotion and why. Emotions play a crucial role in the interaction between characters and the events they are involved in. Until today, no specific corpora that capture such an interaction were available for literature. We aim at filling this gap and present a publicly available corpus based on Project Gutenberg, REMAN (Relational EMotion ANnotation), manually annotated for spans which correspond to emotion trigger phrases and entities/events in the roles of experiencers, targets, and causes of the emotion. We provide baseline results for the automatic prediction of these relational structures and show that emotion lexicons are not able to encompass the high variability of emotion expressions and demonstrate that statistical models benefit from joint modeling of emotions with its roles in all subtasks. The corpus that we provide enables future research on the recognition of emotions and associated entities in text. It supports qualitative literary studies and digital humanities. The corpus is available at
Full-text available
Conference Paper
This paper analyses the development of emotions in different genres of literature. We discover different emotion patterns in adventures, mystery, science fiction, romance and humorous fiction stories. We support our findings with quantitative and qualitative analyses.
Conference Paper
In this paper, we introduce a simple and standalone manual annotation tool for images, audio and video: the VGG Image Annotator (VIA). This is a light weight, standalone and offline software package that does not require any installation or setup and runs solely in a web browser. The VIA software allows human annotators to define and describe spatial regions in images or video frames, and temporal segments in audio or video. These manual annotations can be exported to plain text data formats such as JSON and CSV and therefore are amenable to further processing by other software tools. VIA also supports collaborative annotation of a large dataset by a group of human annotators. The BSD open source license of this software allows it to be used in any academic project or commercial application.
In this article we establish a methodological and theoretical framework for the study of large collections of visual materials. Our framework, distant viewing, is distinguished from other approaches by making explicit the interpretive nature of extracting semantic metadata from images. In other words, one must ‘view’ visual materials before studying them. We illustrate the need for the interpretive process of viewing by simultaneously drawing on theories of visual semiotics, photography, and computer vision. Two illustrative applications of the distant viewing framework to our own research are draw upon to explicate the potential and breadth of the approach. A study of television series shows how facial detection is used to compare the role of actors within the narrative arcs across two competing series. An analysis of the Farm Security Administration–Office of War Information corpus of documentary photography is used to establish how photographic style compared and differed amongst those photographers involved with the collection. We then aim to show how our framework engages with current methodological and theoretical conversations occurring within the digital humanities.