Content uploaded by Nim Dvir
Author content
All content in this area was uploaded by Nim Dvir on Jan 08, 2019
Content may be subject to copyright.
ASIS&T Annual Meeting 2018 797 Visual Presentations
Sticky Words: Evaluation and Optimization of
Information Interactions Based on Linguistic Analysis
Nim Dvir
University at Albany, USA. ndvir@albany.edu
ABSTRACT
This paper describes a novel approach to systematically improve interactions with digital content based solely
on its wording. Following an interdisciplinary literature review, we recognized three key attributes of words that
drive successful interactions: (1) Novelty (2) Familiarity (3) Sentimentality. Based on these attributes, we
developed a model to systematically improve a given content using word frequency, sentiment analysis and
semantic substitution, and by employing computational linguistics and natural language processing (NLP)
tech-niques. We conducted a pilot study (n=416) in which the model was used to formalize evaluation and
optimization of academic titles. A between-group design was used to compare responses to the original
(control) and modified (treatment) titles. Results show that the modified titles had significantly higher scores
for selection, user engage-ment and memorability. Our findings suggest that users’ successful interactions
with digital content is fostered by, and perhaps dependent upon, the wording being used. They also provide
empirical support that engaging content can be systematically evaluated and produced. Implications and future
research directions are discussed.
KEYWORDS
Content strategy, human-computer interaction, user experience (UX), engagement
INTRODUCTION
Providing users with information in a form and format that maximizes its effectiveness is a research question of critical im-
portance. With increasing amount of digital content being published by commercial businesses, governments, healthcare or-
ganizations and private citizens, information interactions and information behavior become more complex (Toms, 2002; Wil-
son, 2000). User engagement (UE) is often used to describe the process of drawing favorable attention, interest and investment
in an interaction between a user and a resource (O’Brien, 2016; O’Brien & Toms, 2008). Given the growing competition for
users’ attention and interest, it is agreed that content must engage. However, there are no clear methods or frameworks for
evaluation, optimization and creation of such content (Dvir & Gafni, 2018; Gafni & Dvir, 2018).
We suggest that the phrasing of the digital content – the words being used – can impact how the information will be consumed,
perceived and used. We propose a conceptual and practical framework to evaluate and improve a given content based on the
identification of reliable and reusable metrics for linguistic analysis and employment of computational linguistic and NLP
techniques.
Following a comprehensive literature review, we conceptualize and operationalize the process of information interaction and
identify three key attributes of digital content that drive successful information interactions: (1) Familiarity - words that are
known and popular, operationalized as having high frequency in popular culture (cultural relevance); (2) Novelty – words that
are rare in the context of the interaction, (operationalized as having low frequency in that context); (3) Emotionality – emotive
words that evoke emotional reaction, operationalized through sentiment polarity (positive, negative, or neutral).
We call these words “sticky words.” We hypothesized that when strategically placed in a given content, these words can in-
crease the motivation to consume it, improve how it is evaluated and lead to better knowledge retention. We conducted a pilot
study in which our model was used to formalize evaluation and optimization of academic titles. The pilot study was guided by
the following research questions: RQ1: Can “sticky words” improve interactions with digital content? RQ2: Can such words
can be systematically, or automatically, evaluated and produced?
METHOD
A randomized between-group design was employed to observe how changes in wording (independent variable) impact infor-
mation interaction and behavior (dependent variable). First, the model was used to create control and treatment groups of
original and modified academic titles. Then, users’ responses were recorded to observe the effect on the interaction.
We collected academic titles from the JSTOR database. We created a corpus of potential “sticky words” using the movie
keyword analyzer of The Internet Movie Database (IMDB.com), which aggregates all keywords assigned to movies. These
represent well-known words frequently used in popular culture. We used Term frequency–Inverse document frequency (TF-
IDF) to find words with high frequency on the list of IMDB keywords (for familiarity), and with low frequency in the collection
of the academic titles (for novelty). Sentiment analysis was used to categorize the words for emotional polarity (positive,
ASIS&T Annual Meeting 2018 798 Visual Presentations
negative, or neutral). We then used semantic and lexical analysis to find synonyms and make replacements of words in the
academic titles, substituting only one word at a time with a semantically equivalent “sticky word.” For example, the title “End
of the library: Organized information and digital ubiquity” became “Death of library: Organized information and digital ubiq-
uity”. We manually verify that the introduction of the new word does not alter the meaning of the text. The result is a dataset
of control and treatment (modified) titles (all titles will be presented in the poster presentation).
We used Qualtric.com to randomly present the titles and to collect responses. Participants (n=416) were recruited using an
email sent to a listserv of undergraduate students in a large research university in the U.S. After providing demographic infor-
mation, each participant was randomly presented with a version of a title, either original (control) or modified (treatment). The
interaction was assessed through questions relating to the different dimensions of information behavior. Specifically (a)
Motivation to consume the content: Participants were asked whether they would like to read the title (b) Evaluation of the
content: Participants were asked to rate the title for interest, value, readability and other factors adopted from the User
Engagement Scale (UES) (O’Brien, Cairns, & Hall, 2018) (c) knowledge retention: Participants answered post-task questions
to test whether they recall the information.
RESULTS
RQ1: Impact of “Sticky words” on the interaction
The effect was examined using univariate analysis of variance (ANOVA), Chi-Square and paired-samples t-tests. Results show
significant difference in favor of the treatment group. The modified titles were significantly ranked higher for motivation to
consume, had higher evaluation scores and higher recall rates. We observed a significant positive correlation between the
treated titles and effective interactions in all dimensions. Thus, the findings indicate that “sticky words” impact information
behavior and lead to a more effective interactions.
RQ2: Can “sticky words” be systematically identified
While we were able to use term frequency, sentiment analysis and semantic substitution to evaluate and replace “sticky words,”
we had to manually confirm that the meaning of the title did not change. The findings indicate the potential to use computational
linguistics to identify factors that predict successful interactions, yet there is still a need to refine our model to achieve full
automation.
CONCLUSION
The implications of this research are twofold. First, our findings suggest that successful interactions with digital content are
fostered by, and perhaps dependent upon, the wording or language being used. Second, we provide empirical support that
engaging content can be evaluated and optimized systematically. We propose that computational linguistics is a useful approach
for studying online information interactions and that further study can result in a broader conceptualization of content strategy
and its evaluation. These empirically based insights can inform the development of digital content strategies, thereby improving
the success of information interactions. Moving forward, the validity, reliability and generalizability of our model should be
tested in various contexts. In future research, we propose to include additional linguistic factors and develop more sophisticated
interaction measures. This research can be used as an important starting point for understanding the phenomenon of digital
information interactions and behavior, the factors that promote and facilitates them, and in the development of a broad frame-
work for systematically evaluation, optimization, and creation effective digital content.
REFERENCES
Dvir, N., & Gafni, R. (2018). When less is more: empirical study of the relation between consumer behavior and information provision on
commercial landing pages. Informing Science: The International Journal of an Emerging Transdiscipline, 21, 019–039.
doi:10.28945/4015
Gafni, R., & Dvir, N. (2018). How content volume on landing pages influences consumer behavior: empirical evidence. In Proceedings of
the Informing Science and Information Technology Education Conference, La Verne, California (pp. 035–053). Santa Rosa, CA: In-
forming Science Institute. doi:10.28945/4016
O’Brien, H. L. (2016). Why Engagement Matters. Cham: Springer International Publishing. doi:10.1007/978-3-319-27446-1
O’Brien, H. L., Cairns, P., & Hall, M. (2018). A practical approach to measuring user engagement with the refined user engagement scale
(UES) and new UES short form. International Journal of Human-Computer Studies, 112, 28–39. doi:10.1016/j.ijhcs.2018.01.004
O’Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology.
Journal of the American Society for Information Science and Technology, 59(6), 938–955. doi:10.1002/asi.20801
Toms, E. G. (2002). Information interaction: Providing a framework for information architecture. Journal of the Association for Infor-
mation Science and Technology, 53(10), 855–862.
Wilson, T. D. (2000). Human information behavior. Informing Science, 3(2), 49–56.
81st Annual Meeting of the Association for Information Science & Technology | Vancouver, Canada | 10 – 14 November 2018
Author(s) Retain Copyright
