Conference PaperPDF Available

Text-analysis after the Machine Learning turn: Evaluation in online book reviews

Authors:

Abstract

Our contribution reports on a research project that addresses distinct research methodologies as well as the epistemic dimension of textual Digital Humanities. In a case study that combines machine learning applications of sentiment detection with more traditional manual and semi-automatic text analysis, such as close reading and annotation, we study evaluative language use in web 2.0 discourses as an example of text production in the context of digitization. Our practical aim in these case studies is to examine how available dictionaries, algorithms, and machine learning procedures work on German textual data extracted from the social web. Addressing the question of how users convey their evaluation of books, we find and generate sets of answers to the question itself using traditional humanities and machine learning approaches, and on the other hand shed light on the role of the distinct research epistemologies as well as our own research practices and attitudes which frame or even shape our findings. One of the central aims for our multidisciplinary mixed-method research design is thus a thorough reflection of the premises underlying our own research practice as different types of actors in the field of textual humanities. Scrutiny will be directed at built-in biases of training corpora used for machine-learning, but also at researchers’ guiding schemata for scientific/scholarly action: these are mutually determined by and determine reality and humanity, showing in research practice – as types of research questions, argumentation, interpretation, and methodology. Our methods consist in a range of tools for reflective thinking, including meta-annotation of the (semi-)automatic and manual tags and memos for documenting observations. On this level, we will address the following questions: 1. Theoretical frameworks: What understanding of language and communication motivate the different approaches? What assumptions are made about the world, reality and humanity? 2. Operationalization: What, if any, linguistic and literary categories and theories are being operationalized? What covert and overt decisions inform existing analytical practices? 3. Results and interpretation: What rationale is used to discuss and interpret results? How do interpretations of results tie in with the respective theoretical frameworks? What open questions are foregrounded? What role and importance are given to the interpretation of data within the overall research projects? How is subjectivity discussed? 4. Research practice: What explicit and implicit rationale motivates the employment of particular research processes? Using the empirical research on web 2.0 discourses throughout its stages for our enquiry, we examine the underlying epistemologies that our team brings to the table, extrapolating from here to our field of textual DH generally. Our goal is not to find the best method, but to reveal underlying worldviews and epistemologies that guide the analysis through principled reflective thinking.
Text-analysis after the Machine
Learning turn: Evaluation in
online book reviews
Thomas C. Messerli & J. Berenike Herrmann
Digital Humanities Lab, Basel University
© https://goo.gl/images/hcXDhJ
SNSF: Digital Lives
“Research Approaches and Practices in the Text-based Digital Humanities.
Analysis of Evaluation Practices after the Machine Learning Turn”
University of Basel (Herrmann/Messerli)
University of Zurich/ZHAW (Bubenhofer/Knuchel)
Impact of digitization on …
Humanities and social sciences
Social and subjective dimensions of Lebenswelt
http://www.forschungslogiken.net/en/
Literary Studies
Corpus Linguistics
Computational Linguistics
Literature
on the social web
Evaluation online
“We all love talking about books,
recommending them to our friends or sharing
our criticism when we did not like them.
LovelyBooks is the place on the internet where
all of this is possible the home of book lovers
and reading aficionados. Nice that you are
here!“
(translated from: https://www.lovelybooks.de/info/ueberuns/)
Online review platform Lovelybooks.de
Largest German-language book
community (self-declared)
340,000 registered members (2019,
Media data)
1.2 mio. reviews of 181,000 books
(77,000 authors)
2.7 mio. texts (short opinions,
comments, etc.),
60,000 new texts per month
Community (book shelves, wish
lists, currently reading, followers,
comments, likes)
A new type of literary field
Lay Critic
= User
Simple books? Simple readers?
Traditional criticism
challenges and pleasures of
literature-as-art
tool for honing crucial sense-
making capacities
enabling an integrated sensual
intellectual personal
development (judicium
sensitivum, Wolff, 1738)
Web 2.0 users
easy gratification, popular genres
(economy of attention, Franck, 2004)
affirmative bias: lack of deep &
discerning reading engagement (Ingold,
2014)
trend towards ‘digestible’ documentary
& authentic literature (Röhricht, 2016):
content bias, neglect of formal criteria
subjectivity ofeffect aesthetic” (taste)
Complex literature <--> complex criticism
Simple literature <--> simple criticism
Complex literature <--> simple criticism
Building a Complex Model
What do web 2.0 readers actually do
when judging literary value?
What underlying values and value
systems emerge when we analyse
patterns?
Do readers apply aesthetic and ethical
premises?
Do they prioritize content over formal
criteria?
What variables interact?
Genre
Theme
Quantitative evaluation
User/community-related
variables
Literary Evaluation
To select functionally, describe, also interpret and thus to evaluate
artful literary ‘objects‘ (the literary ‘text‘)“ (Albrecht, 2001)
“The term axiological value refers to the scale by use of which an
object or a property of an object is deemed ‘valuable, i.e. why it is
recognisable as a value. (Heydebrand/Winko, 1996: 40, our translation)
Heydebrand, R. von & Winko, S. (1996). Einführung in die Wertung von Literatur : Systematik -Geschichte -
Legitimation. Paderborn, Zürich [etc.]: Schöningh.
Worthmann, F. (2004) Literarische Wertungen. Vorschläge für ein deskriptives Modell, Wiesbaden.
Albrecht, W. (2001). Literaturkritik. Stuttgart/Weimar
Elements of a review
user name
User name
Date of review
Book cover, title, author
Star rating (n/5)
Short opinion
Title of review
Review
text formatting, lists, URLs, pictures, links
to Youtube
Status, book shelves, wishlist
Tags (open)
Social network: comments, likes by
other users
Elements of a review
I liked the atmosphere of the bright nights.
That was very well described and captured.
However, all in all I found the book to be firstly a
little too melancholy, and secondly – which was
of more importance when awarding a star
rating some of the events were not fully
comprehensible.
I always find this especially regrettable, as it is
an aspect I find very important.
The persons (characters) were done
successfully, as was the atmosphere as
mentioned above already.
But the things that people do or do not do are
especially important to me and I am always
very confused/irritated when…
Evaluation
I liked
very well (described and captured)
a little too (melancholy)
not fully (comprehensible)
especially regrettable
especially important to me
Criteria for evaluation
1. Description
described, captured
2. Coherence/Cohesion
(I am always very) confused/irritated
(not fully) comprehensible
3. Plot/plot elements
the things that people do…
persons (characters)
4. Effect on readers
atmosphere (liked, done successfully)
(a little too) melancholy
Weight of criteria
of more importance when awarding a star rating
I always find / I am always
an aspect I find very important
especially important to me
I liked the atmosphere of the bright nights.
That was very well described and captured.
However, all in all I found the book to be firstly a
little too melancholy, and secondly – which was
of more importance when awarding a star
rating some of the events were not fully
comprehensible.
I always find this especially regrettable, as it is
an aspect I find very important.
The persons (characters) were done
successfully, as was the atmosphere as
mentioned above already.
But the things that people do or do not do are
especially important to me and I am always
very confused/irritated when…
Analysis
Corpus of book reviews (approx. 1.3 mio) across 15 genres
Language use
Lexis, POS (adjectives, nouns, verbs)
Sentiment analysis
Star ratings
+
Emic/etic criteria, hierarchies
Values and value systems
e.g. realism, aesthetics, ideology, hedonism, effects on readers
Method: cwb
[lemma=“gut” %cd][]{0,4}[pos=“N.*”]
CWB example
Search:
[(lemma="arm” %cd)&(pos="N.*")][]{0,4}[lemma="Mutter"]|[lemma="Mutter"][]{0,4}[(lemma="arm" %cd)&(pos="N.*")]
Result:
CWB Metadata
Evaluation vocabularies:
adjectives
gut (‘good’), beeindruckend (‘impressive’)
gut
(‘good’)
2010609
einfach
(‘simple’)
816011
wirklich
(‘real’)
790658
erst
(‘first’)
701298
spannend (‘suspenseful’)
673284
gross (‘big/large’)
495180
neu (‘new’)
493882
schön
(‘beautiful’)
489403
toll
(‘great’)
471198
klein
(‘little/small’)
433450
schnell (‘quick/fast’)
366647
richtig
(‘right/correct’)
365270
ganz
(‘whole’)
302450
kurz
(‘short’)
298817
interessant
(‘interesting’)
292697
jung
(‘young’)
286988
eigen (‘own’)
276662
genau
(‘exact/precise’)
271215
weit
(‘far’)
268122
leicht
(‘light/easy’)
258337
letzt
(‘last’)
257569
alt (‘old’)
244466
nah (‘near’)
240912
zweit
(‘second’)
236579
absolut
(‘absolute’)
230178
klar
(‘clear’)
218683
recht (‘right/appropriate’)
194207
LOBO v1.0
Tota l N= * 439,912,009 tokens
Tota l N= *187,275 Lemmata
Software:
Adjectives
raw Frequency
LogDice
Buch (‘book’)
Buch (‘book’)
Freundin
(‘friend-F’)
Freundin
(‘friend-F’)
Freund (‘friend
-M’)
Geschichte
(‘story/history’)
Geschichte
(‘story/history’)
Freund (‘friend
-M’)
Schreibstil
(‘writing style’)
Charakter
(‘character’)
Charakter
(‘character’)
Schreibstil
(’writing style’)
Teil
(‘part’)
Teil
(‘part’)
Idee
(‘idea’)
Band (‘volume/band’)
Band (‘volume/band’)
Idee
(‘idea’)
Ende (‘ending/end’)
Ende (‘ending/end’)
Roman (‘novel’)
Autorin
(‘author-F’)
Rezension
(‘review’)
Roman (‘novel’)
Story (‘story/plot’)
Handlung
(‘action’)
Software:
LOBO v1.0
Tota l N= * 439,912,009 tokens
Tota l N= *187,275 Lemmata
Collocations gut (‘good’) + N
gut (‘good’) by rating
per 100k words
Rating in Stars
0
100
200
300
400
500
600
5 4 3 2 1
beeindruckend (‘impressive’) by rating
per 100k words
Rating in Stars
0
1
2
3
4
5
6
7
8
54321
beeindruckend (‘impressive’) by genre
per 100k words
0
5
10
15
20
25
30
35
biografie
klassiker
historische-romane
gedichte-drama
romane
comic
sachbuch
nan
science-fiction
fantasy
jugendbuch
kinderbuch
krimi-thriller
liebesroman
erotische-literatur
humor
biography
LogDice
Debüt
(‘debut’)
(‘story/history’)
Persönlichkeit
(‘personality’)
Lebensgeschichte
(‘life story’)
Weise (‘way’)
Werk
(‘oeuvre/work’)
(‘oeuvre/work’)
Kulisse
(‘setting’)
(‘character’)
Schilderung
(‘description’)
(‘debut’)
Leistung
(‘achievement’)
Debütroman
(‘first novel’)
(‘personality’)
Landschaft
(‘landscape’)
Leseerlebnis
(‘reading experience’)
(‘writing style’)
Biographie
(‘biography’)
(‘author-F’)
Bild (‘image/picture’)
LOBO v1.0
Tota l N= * 439,912,009 tokens
Tota l N= *187,275 Lemmata
Collocations beeindruckend
(‘impressive’) + N
Beyond adjectives: Comparisons
ist … so [ADJ] wie (‘is … as [ADJ] as’)
The story itself is as old as stone age.
Die Story selbst <ist so alt wie> die Steinzeit .
The story of the two is as touching as few others.
Die Geschichte der Beiden <ist so berührend wie> kaum eine andere.
full of suspense like a novel and yet is as instructive as non-fiction
spannend und unterhaltsam wie ein Roman und <ist dennoch so lehrreich wie> ein Sachbuch
Sentiment: Senti WS
SentimentWortschatz SentiWS, Uni Leipzig, publicly available
V 2.0 2018-10-19 (3rd version)
Positive and negative polarity, within [-1;1]; + POS + word forms
1650 positive and 1800 negative words
R. Remus, U. Quasthoff & G. Heyer: SentiWS -a Publicly Available German-language Resource for Sentiment
Analysis. In: Proceedings of the 7th International Language Ressources and Evaluation (LREC'10), pp. 1168-
1171, 2010 http://www.lrec-conf.org/proceedings/lrec2010/pdf/490_Paper.pdf
Word
Freq
Ende (’end/ending’)
619137
klein
(‘small/little’)
433488
leider
(‘unfortunately’)
400963
kurz
(’short’)
299216
alt (‘old’)
244466
schwer
(‘difficult/heavy’)
192385
Problem (‘problem’)
161993
Tod (‘death’)
154325
verlieren
(‘lose’)
137760
fehlen
(‘be absent’)
131283
Word
Freq
gut ‘(good’)
2010663
einfach
(‘simple’)
834497
spannend (‘suspenseful’)
673284
mögen
(‘like’)
546979
gross (‘big/large’)
495440
neu (‘new’)
494049
schön
(‘beautiful’)
490575
toll (‘great’)
471198
Liebe (‘love’)
373998
schnell (‘fast/quick’)
366647
+-
Sentiment: Senti WS
0
1000
2000
3000
4000
5000
6000
7000
54321
per 100k words
+
-
++++
--
-
-
ratings
Sentiment: Senti WS
Personal Pronouns
ich, mich, mir, mein, wir, uns, unser, du, dich, dir, dein
2nd
person
singular
pronouns by genre
per 100k words
0
20
40
60
80
100
120
140
gedichte-drama
liebesroman
erotische-literatur
jugendbuch
sachbuch
biografie
romane
nan
humor
kinderbuch
krimi-thriller
fantasy
science-fiction
comic
klassiker
historische-romane
Tot a l
poetry/drama
1st
person
plural
pronouns by genre
per 100k words
0
50
100
150
200
250
300
350
400
sachbuch
kinderbuch
gedichte-drama
biografie
science-fiction
humor
romane
nan
comic
klassiker
jugendbuch
fantasy
historische-romane
liebesroman
erotische-literatur
krimi-thriller
Tota l
non-fiction
children’s literature
1st
person pronouns by rating
88%
90%
92%
94%
96%
98%
100%
1 2 3 4 5
1st person plural
1st person singular
plural
singular
Results
How do lay reviews evaluate?
Evaluation
Establishing scales
Weight of criteria
Indicators of evaluation
Adjectives and beyond
Pronouns
Distribution of evaluative language
Genre
Rating
Sentiment
Conclusions
Do lay reviews signify a decline in
intellectual scrutiny?
It‘s a bit more complex!
Democratization of literary criticism
Variability and diversity
Potential for personal insight and
development, community dimension
Data-driven approach to literary
values
Classification
(stars/sentiment)
Description (lexis, POS,
sentiment, subj/obj)
àMachine Learning
thomas.messerli@unibas.ch
Thank you
for your attention.
twitter
@pointyear
ResearchGate has not been able to resolve any citations for this publication.
Einführung in die Wertung von Literatur : Systematik -GeschichteLegitimation
  • R Heydebrand
  • S Winko
Heydebrand, R. von & Winko, S. (1996). Einführung in die Wertung von Literatur : Systematik -GeschichteLegitimation. Paderborn, Zürich [etc.]: Schöningh.
Literarische Wertungen. Vorschläge für ein deskriptives Modell
  • F Worthmann
Worthmann, F. (2004) Literarische Wertungen. Vorschläge für ein deskriptives Modell, Wiesbaden. Albrecht, W. (2001). Literaturkritik. Stuttgart/Weimar raw Frequency LogDice Buch ('book') Buch ('book')
friend-F') Freundin ('friend-F')
  • Freundin
Freundin ('friend-F') Freundin ('friend-F')
writing style') Charakter ('character')
  • Schreibstil
Schreibstil ('writing style') Charakter ('character')
character') Schreibstil ('writing style')
  • Charakter
Charakter ('character') Schreibstil ('writing style')
part') Teil ('part')
  • Teil
Teil ('part') Teil ('part')
novel') Autorin ('author-F')
  • Roman
Roman ('novel') Autorin ('author-F')
novel') Lebensgeschichte ('life story')
  • Roman
Roman ('novel') Lebensgeschichte ('life story')
image/picture') Debütroman ('first novel')
  • Bild
Bild ('image/picture') Debütroman ('first novel')
writing style') Biographie ('biography')
  • Schreibstil
Schreibstil ('writing style') Biographie ('biography')