Content uploaded by Annelien Smets
Author content
All content in this area was uploaded by Annelien Smets on Mar 27, 2022
Content may be subject to copyright.
RESEARCH ARTICLE
Serendipity in the city: User evaluations of urban
recommender systems
Annelien Smets | Jorre Vannieuwenhuyze | Pieter Ballon
imec-SMIT, Vrije Universiteit Brussel,
Brussels, Belgium
Correspondence
Annelien Smets, imec-SMIT, Vrije
Universiteit Brussel, Pleinlaan 9, 1050
Brussels, Belgium.
Email: annelien.smets@vub.be
Abstract
The contemporary city is increasingly being labeled as a smart city consisting
of both physical and virtual spaces. This digital augmentation of urban life sets
the scene for urban recommender systems to help citizens dealing with the
abundance of digital information and corresponding choice overload, for
example, by recommending the best place to have dinner based on your per-
sonal profile. There are, however, concerns that this kind of algorithmic filter-
ing could lead to homogenization of urban experiences and a decline of social
cohesion among citizens. To overcome this issue, scholars increasingly encour-
age the introduction of serendipity in all types of recommender systems. None-
theless, it remains unclear how this can be achieved in practice. In this work,
we study user evaluations of serendipity in urban recommender systems
through a survey among 1,641 citizens. More specifically, we study which
characteristics of recommended items contribute to serendipitous experiences
and to what extent this increases user satisfaction and conversion. Our results
align with findings in other application domains in the sense that there is a
strong relation between the relevance and novelty of recommendations and
the corresponding experienced serendipity. Moreover, serendipitous recom-
mendations are found to increase the chance of users following up on these
recommendations.
1|INTRODUCTION
In contemporary cities, activities and experiences are
increasingly shaped by digital urban media (de
Waal, 2013) such as urban recommender systems
(Quijano-S!
anchez et al., 2020). Citizens use these
systems to help choose among possible destinations and
activities (e.g., TripAdvisor), places to eat or rest
(e.g., Airbnb) or navigating through the urban environ-
ment (e.g., Waze). These urban recommender systems
are an important example of real-time merging of
digital and physical public space to achieve efficiency
and productivity. As such, they are argued to contribute
to the so-called “smart city”(van der Graaf &
Ballon, 2019).
Although urban recommender systems help citizens to
cope with the abundance of available digital information,
they are often built on the premise that users are merely
looking for information closely matching their own pro-
files (Iaquinta et al., 2008; Jannach & Adomavicius, 2016).
For example, recently it has been shown that Google Maps
guides people to the very same types of restaurants,
museums or activities over and over again based on their
demographic profiles and search history (Smets et al.,
2019). As a consequence of this premise, concerns are
raised that recommender systems may create urban “filter
bubbles”(see Pariser, 2011) by merely exposing users to
predictable, popular and homogeneous content rather
than challenging their views with serendipitous encoun-
ters (Foth, 2017; Zuiderveen Borgesius et al., 2016).
Received: 13 November 2020 Revised: 14 June 2021 Accepted: 1 July 2021
DOI: 10.1002/asi.24552
J Assoc Inf Sci Technol. 2021;1–12. wileyonlinelibrary.com/journal/asi © 2021 Association for Information Science and Technology. 1
Annelien Smets, Jorre Vannieuwenhuyze & Pieter Ballon
imec-SMIT, Vrije Universiteit Brussel
The contemporary city is increasingly being labeled as a smart city consisting of both physical and
virtual spaces. This digital augmentation of urban life sets the scene for urban recommender systems to
help citizens dealing with the abundance of digital information and corresponding choice overload, for
example, by recommending the best place to have dinner based on your personal profile. There are,
however, concerns that this kind of algorithmic filter- ing could lead to homogenization of urban
experiences and a decline of social cohesion among citizens. To overcome this issue, scholars
increasingly encourage the introduction of serendipity in all types of recommender systems.
Nonetheless, it remains unclear how this can be achieved in practice. In this work, we study user
evaluations of serendipity in urban recommender systems through a survey among 1,641 citizens. More
specifically, we study which characteristics of recommended items contribute to serendipitous
experiences and to what extent this increases user satisfaction and conversion. Our results align with
findings in other application domains in the sense that there is a strong relation between the relevance
and novelty of recommendations and the corresponding experienced serendipity. Moreover,
serendipitous recommendations are found to increase the chance of users following up on these
recommendations.
To cite this article: Smets, A., Vannieuwenhuyze, J., & Ballon, P. (2021). Serendipity in the
city: User evaluations of urban recommender systems. Journal of the Association for
Information Science and Technology, 1–12. https://doi.org/10.1002/asi.24552
Serendipity in the city: User
evaluations of urban recommender
systems
Serendipity refers to “what happens when we, in
unplanned ways, encounter resources that we find inter-
esting”(Björneborn, 2017, p.2). As such, it has been
argued that serendipitous encounters in the city are not
only key factors for cities' economic and innovative
growth (Wood & Landry, 2008), but that they are also
central to social bonding and urban trust as described by
Jacobs' (1961) notion of “well-used streets”. As a result,
urban recommender systems ignoring serendipitous rec-
ommendations may form a threat to the urban ecosystem
(McQuire, 2017). This threat of people not being exposed
to and excluded from spontaneous interaction with the
diversity of cities and their inhabitants is a timely issue
that needs urgent attention (Smets et al., 2019).
Recently, there has been an increased focus on seren-
dipity in the domain of recommender systems. Nonethe-
less, this line of research still shows some gaps. Firstly,
serendipity in recommender systems has rarely been
investigated within an urban context. Over the last
decade, the use of recommender systems and their failure
to introduce serendipity has mainly been discussed in the
context of mere online (social media) platforms (Reviglio,
2019). Only a few preliminary urban digital information
systems have been developed introducing serendipity into
urban recommender systems, as a merger between a digi-
tal and a physical environment rather than an entirely
digital environment. These few examples include applica-
tions for urban navigation (Delva et al., 2020; Ge
et al., 2017; Li & Tuzhilin, 2019; Shepard, 2011) or appli-
cations to connect strangers in public places (Paulos &
Goodman, 2004). Research on serendipity has, however,
demonstrated that serendipity evolves differently in dif-
ferent contexts inducing the need to further investigate
serendipity in urban recommender systems in particular
(Lutz et al., 2017; Olshannikova et al., 2020; Sun
et al., 2011).
Secondly, the small amount of existing studies about
serendipity in urban recommender systems rarely focus
on user experience and evaluation in the field. The
majority of these studies rely on offline lab experiments
for design optimization (Kotkov, Veijalainen, &
Wang, 2016). Such experiments can be useful in choosing
candidate algorithms, but often take place in artificial set-
tings in which specific recommender systems are
imposed on test subjects. In order to circumvent the limi-
tations of such lab experiments, large scale field studies
are required to investigate user feedback in daily life situ-
ations (Kotkov, Veijalainen, & Wang, 2016). Due to the
involvement of actual users those field studies are costlier
and therefore often neglected (Silveira et al., 2019).
This study aims to fill these gaps by investigating user
evaluations of serendipity in urban recommender sys-
tems. More specific, this study addresses the question
which characteristics of urban recommendations lead to
serendipity experiences and to what extent this increases
user satisfaction and conversion (i.e., the capacity of the
recommender system to convince users to follow up on
the recommendations). In this way, the results of this
study provide a first insight into user evaluations of ser-
endipity in urban recommender systems.
The paper is structured as follows. The next
section provides an overview of the literature on urban
recommender systems and the role of serendipity in those
systems. This overview will lead to a set of research ques-
tions under study in this work. Section 3 introduces a sur-
vey among 1,641 citizens about experienced serendipity,
user satisfaction and conversion in urban contexts. Sec-
tion 4, subsequently, elaborates on the results of the ana-
lyzed survey data. Finally, Section 5 ends the paper with
a discussion of the results.
2|THEORETICAL FRAMEWORK
2.1 |Serendipity
Serendipity in digital environments has been studied
from a wide range of perspectives, each emphasizing
their own focus and assumptions (Reviglio, 2019). In
information science and information behavior research,
emphasis is usually put on process models explaining the
occurrence of serendipity experiences (Erdelez, 2004;
Erdelez & Makri, 2020; Lutz et al., 2017; Makri &
Blandford, 2012). Various authors, for example, identified
several contextual factors related to the user, information,
tasks and the environment that influence the process of
information encountering (Erdelez & Makri, 2020; Jiang
et al., 2015). Such contextual factors are also adopted in
the field of information system design, in which they are
considered precipitating conditions increasing the likeli-
hood of serendipity (Björneborn, 2017; McCay-Peet &
Toms, 2011). Several studies have been conducted trying
to assess these conditions and the extent to which they
contribute to serendipity. Here, however, a mere distinc-
tion is usually made between personal cognitive and
behavioral antecedents (Lutz et al., 2017) versus environ-
mental factors or characteristics (McCay-Peet &
Toms, 2011). Those environmental factors consist of
various aspects that have been categorized by Björneborn
(2017) as three key affordances: diversifiability,
traversability and sensoriability. These affordances
should be considered as building blocks when designing
environments that facilitate serendipity, and respectively
refer to the ability of the environment to allow a diversity
of contents, to be traversable and to be perceivable by the
senses.
2SMETS ET AL.
In the context of recommender systems, environmen-
tal affordances refer to characteristics of the system itself.
That is the diversity of the recommended items, the navi-
gation and interactivity of the system and the user inter-
face design. Most of the work on serendipity in
recommender systems has, however, mainly been dealing
with the recommended items themselves. In this strand
of the literature, serendipity is most commonly
considered as a compound concept consisting of three
characteristics of recommended items: relevance, novelty
and diversity (Chen et al., 2019; Kotkov, Wang, &
Veijalainen, 2016; also see Figure 1).
Relevance refers to recommended items that users like
or are interested in (Iaquinta et al., 2008; Kotkov,
Wang, & Veijalainen, 2016; Maksai et al., 2015) and is
usually measured by the accuracy of predicted user rat-
ings for unseen items. It is an important aspect of seren-
dipitous recommendations because, by definition,
serendipity refers to encounters that are relevant to the
user. Nonetheless, sole focus on relevance may lead to fil-
ter bubbles because it precludes unexpectedness (Kotkov,
Wang, & Veijalainen, 2016).
Indeed, serendipity also requires unexpectedness,
which may be facilitated by including novelty and diver-
sity in recommended items (Ge et al., 2010; Kotkov,
Veijalainen, & Wang, 2016; Tacchini, 2012). Novelty
refers to items that are unknown to the user either
because they are (1) novel to the system, (2) forgotten by
the user, (3) unknown to the user or (4) unrated by the
user (Iaquinta et al., 2008; Kotkov, Wang, &
Veijalainen, 2016). Diversity refers to the variability in
recommended items a user receives (Kotkov, Wang, &
Veijalainen, 2016). It was already found by Chen
et al. (2019) that both novelty and diversity are important
antecedents of unexpectedness, which in turn, affects
experiences of serendipity. Nonetheless, the overall influ-
ence of diversity on serendipity was not confirmed by
their study. The availability of a diverse set of items is,
however, considered to be a key environmental
affordance to foster serendipity (Björneborn, 2017).
The hypothesized influence of relevance, novelty and
diversity on experienced serendipity brings us to our first
research question:
RQ1. How do relevance, novelty and diver-
sity affect users' experienced serendipity in
urban recommender systems?
2.2 |User satisfaction and conversion
The second goal of this study is to investigate whether
experiences of serendipity in urban recommender sys-
tems also lead to higher user satisfaction and, conse-
quently, higher user conversion rates (i.e., the ability of
the recommender system to persuade users to actually
follow up on the recommendations). After all, it has been
assumed that sole focus on relevance, in contrast to ser-
endipity, does not optimize user satisfaction because
users do not appreciate lists with very similar items
(Chen et al., 2019; De Gemmis et al., 2015; Kotkov,
Veijalainen, & Wang, 2016; Lutz et al., 2017; Said et al.,
2013; Zhang et al., 2012). User satisfaction, in turn, has
been shown to increase user conversion (Chen et al.,
2019; Venkatesh et al., 2012).
Within the existing literature, it has already been
shown that diversity positively correlates with user satis-
faction (De Gemmis et al., 2015; Kotkov, Veijalainen, &
Wang, 2016; Kunaver & Požrl, 2017). In the context of
serendipity, however, these findings contradict with the
results of previously mentioned studies where only small
relations were found between diversity and experiences
of serendipity (Chen et al., 2019).
Mixed results have been found for the relationship
between novelty and user satisfaction (Chen
et al., 2019; Ekstrand et al., 2014). Novel items do not
necessarily positively correlate with user satisfaction or
conversion because novelty could also decrease users'
trust in the capabilities of the system (Ekstrand
et al., 2014).
However, given the call for contextual differentiation
in serendipity research and the current limited focus on
serendipity in urban recommender systems, this study
aims to further investigate the relation between serendip-
ity antecedents and user satisfaction. In sum, we hypoth-
esize that relevance, novelty and diversity affect user
satisfaction and, subsequently, user conversion through
FIGURE 1 We hypothesize that the relevance, novelty and diversity of recommended items in urban recommender systems affect user
satisfaction and, subsequently, user conversion through experienced serendipity
SMETS ET AL.3
experienced serendipity (see Figure 1). This brings us to
the following research questions:
RQ2. Do relevance, novelty and diversity
influence users' satisfaction in urban recom-
mender systems and can this influence be
explained by experienced serendipity?
RQ3. Do relevance, novelty and diversity
influence user conversion in urban recom-
mender systems and can this influence be
explained by experienced serendipity and user
satisfaction?
2.3 |Contextual differentiation
The third goal of this study is to investigate whether
experienced serendipity also depends on the system's
domain or user's needs. After all, it has already been
shown that users might have different needs for serendip-
ity in different recommendation scenarios (Ekstrand
et al., 2014; Kaminskas & Bridge, 2016; McCay-Peet, 2014;
Sun et al., 2011). Indeed, serendipity in recommender
systems has already been studied in various contexts such
as e-commerce (Chen et al., 2019; Lutz et al., 2017),
movies (Kotkov et al., 2018), music (Matt et al., 2014;
Zhang et al., 2012) and social networking sites (Lutz
et al., 2017), all leading to variable results. As a result,
acknowledging that the urban environment is eminently
heterogeneous, studying serendipity in urban recom-
mender systems also requires to take a contextual differ-
entiation into account. This brings us to our final
research question:
RQ4. Does the impact relevance, novelty and
diversity on experienced serendipity in urban
recommender systems depend on the context
of use?
3|METHODS
3.1 |Sample
Most existing studies on serendipity experiences with rec-
ommender systems adopt an experimental approach in
which test persons are confronted with an artificial rec-
ommender system after which they are immediately
asked for their experiences (Ekstrand et al., 2014; Pu
et al., 2011). However, especially in the context of offline
activities, such experimental studies create settings that
may significantly deviate from regular situations in daily
life in which people use recommender systems. As an
alternative, we decided to use a survey with retrospective
questions about actual behavior and experiences in real-
life situations.
We investigated the user evaluations of serendipity in
urban recommender systems through the Smart City
Meter 2020. This is an annual survey in Flanders and
Brussels, Belgium, about citizens' opinions, attitudes
and behaviors in the context of smart cities. The data
were collected between March 1 and April 30, 2020
among people recruited through an online panel of a pri-
vate market research agency. This panel had been col-
lected and maintained over the years through various
projects of the agency. A stratified sample was taken from
this panel according to gender, age and place of residence
(Brussels, Antwerp, Ghent, other large cities, small
towns, municipalities). The size of the strata was deter-
mined by the distribution of these variables among the
Brussels and Flemish population. In addition, strata size
was adjusted according to the response rate within each
stratum based on previous experience of the agency.
However, all drawn panel members were invited by
email to complete the questionnaire in the same way and
with the same number of contact attempts.
In total, 1,641 eligible panel members responded to
the questionnaire. Because of the disproportional strati-
fied sampling strategy, the realized sample more or less
followed the population distribution of gender, age and
place of residence. Nonetheless, because of panel con-
straints, people under the age of 30 and over the age of
70 were slightly underrepresented. For that reason, based
on the population distributions, the respondents were
assigned analysis weights. Further analysis also showed
that the sample included both higher and lower educated
people and voters of all relevant political parties.
3.2 |Variables
In order to allow for contextual differentiation (cf. RQ4)
the respondents were randomly divided into two groups.
The first group was confronted with questions about rec-
ommender systems for Catering in the city (restaurants
and bars). The second group, in turn, was confronted
with questions about recommender systems for general
Activities in the city. The respondents were firstly asked
how often they use recommender systems (websites or
apps like Google, TripAdvisor, …) to find new catering
stores or to find things to do in the city respectively.
Respondents who indicated to never use such websites or
apps were not asked any further questions about these
recommender systems and were forwarded to the next
questionnaire section. All other respondents, in contrast,
4SMETS ET AL.
got follow-up questions about serendipity, satisfaction
and conversion in these recommender systems.
Unfortunately, within the existing literature, exam-
ples of measurement instruments for serendipity experi-
ences are scarce. Additionally, the few existing
operationalizations are also quite diverse. Some use sev-
eral agree-disagree statements for measuring specific
dimensions of serendipity like perceived recommenda-
tion diversity (Knijnenburg et al., 2012). Others devel-
oped item sets for measuring experiences about
serendipity affordances based on the work of Björneborn
(McCay-Peet & Toms, 2011) or implemented a set of
questions allowing test persons to compare different rec-
ommender systems (Ekstrand et al., 2014). Some investi-
gated the potential of survey questions for measuring a
various range of serendipity definitions (Kotkov
et al., 2018) or instantaneous experiences of serendipity
(Lutz et al., 2017). Because of space constraints, however,
we based our work on the survey items used by Chen
et al. (2019), who adopted a short single-item version of
the ResQue evaluation framework for recommender sys-
tems (see Pu et al., 2011).
In order to measure relevance of recommended items
respondents were asked how often they get recommenda-
tions that suit them well (see the Appendix). For measur-
ing novelty, they were asked how often they get
recommendations they did not know yet. For (lack of)
diversity, they were asked how often they get the same
kind of recommendations. For experienced serendipity,
respondents were asked how often they find themselves
pleasantly surprised by the recommendations in these
systems. User satisfaction was measured by a question
about how satisfied respondents generally are with the
recommendations they usually get. User conversion was
measured by a question about how often they actually
follow the provided recommendations in such recom-
mendation systems. Respondents could provide answers
to all these questions through 5-point Likert scales. The
Likert scales were treated as continuous variables in all
analyses below (see Table 1).
3.3 |Analysis
Given that our theoretical model (Figure 1) assumes an
indirect effect of relevance, novelty and diversity on rec-
ommender system conversion through experienced ser-
endipity and user satisfaction, we used mediation
analysis to model the data. Mediation analysis refers to
the investigation of direct and indirect effects of a set of
exogenous independent variables on the dependent vari-
ables through mediator variables (MacKinnon, 2008).
Within the existing literature, mediation analysis usu-
ally starts from an investigation of the total effect of the
TABLE 1 The questionnaire included questions about serendipity, satisfaction and conversion of recommender systems in urban
environments. The bivariate correlations between the serendipity antecedents, satisfaction and conversion are moderate to high, except for
diversity
Mean SD
Correlations
Relevance Novelty Diversity Serendipity Satisfaction
Catering group
Relevance 3.25 .81
Novelty 3.40 .83 .41
[<.001]
Diversity 3.35 .80 .05
[.140]
Serendipity 3.03 .76 .58
[<.001]
.44
[<.001]
.00
[.900]
Satisfaction 3.66 .71 .59
[<.001]
.41
[<.001]
.01
[.794]
.46
[<.001]
Conversion 2.92 .74 .50
[<.001]
.37
[<.001]
.19
[<.001]
.46
[<.001]
.35
[<.001]
Activities group
Relevance 3.33 .80
Novelty 3.16 .78 .40
[<.001]
Diversity 3.54 .78 !.07
[.047]
!.10
[.007]
Serendipity 2.82 .79 .54
[<.001]
.48
[<.001]
!.17
[<.001]
Satisfaction 3.74 .71 .64
[<.001]
.37
[<.001]
!.09
[.012]
.46
[<.001]
Conversion 2.62 .83 .49
[<.001]
.40
[<.001]
!.06
[.066]
.57
[<.001]
.37
[<.001]
Note: p-values between square brackets. All responses were collected on 5-point Likert-scales. Question wording and response scales can be found in the
appendix.
SMETS ET AL.5
independent variables on the dependent variables (see
Baron & Kenny, 1986). Subsequently, the effect of the
independent variables on the dependent variables is mea-
sured again but controlling for the mediator variables.
This allows for discriminating between the direct and
indirect effect of the independent variables on the depen-
dent variables. Nonetheless, it is advised to model all
effects simultaneously in one single analysis model
(Rucker et al., 2011). For that reason, we conducted a
path analysis were paths were defined along the theoreti-
cal model in Figure 1 additional to direct effects of the
serendipity antecedents relevance, novelty and diversity
on the dependent variables user satisfaction and conver-
sion. The model was estimated in R using the lavaan
package (Rosseel, 2012). Multi-group estimation was used
to distinguish between the Catering scenario and the
Activities scenario. Parameter estimates were obtained by
maximum likelihood estimation with robust Huber-
White standard errors to avoid problems of non-
normality.
Note that we do not correct p-values for multiple test-
ing since this research is exploratory. The interpretation
of results is mainly based on estimated effect sizes rather
than p-values. After all, p-values are bad measures of
effect sizes or the importance of results (Amrhein
et al., 2019; Greenland et al., 2016; Wasserstein &
Lazar, 2016). Moreover, adjustments for multiple testing
increase type II errors for non-null associations and are
calculated based on arbitrarily chosen numbers of tests
(Feise, 2002; Rothman, 1990).
4|RESULTS
4.1 |Descriptive analyses
On average, the respondents provided similar answers to
all survey items (see Table 1). Indeed, they were mostly
undecided about whether recommender items are usually
relevant, novel and diverse, whether they experience ser-
endipity, are satisfied by the recommendations and adopt
to the recommendations, because the observed means of
all items are close to the center of the scale, that is,
3. Nonetheless, except for conversion, the responses were
slightly skewed to the positive side of the response scales
in both the Catering as well as the Activities respondent
groups. Additionally, the spread of the responses was
more or less similar across all items in both experimental
groups, as shown by the observed standard deviations.
When considering the bivariate correlations between
the six items in both contexts, the data firstly reveal rela-
tively strong relations between relevance, novelty and
experienced serendipity. Diversity, in contrast, is very
weakly correlated with experienced serendipity as well as
with both other antecedents. This suggests that, in con-
trast to relevance and novelty, diversity in recommenda-
tions does not cause users to experience serendipity.
Regarding satisfaction with the outcomes of the
recommender systems, the data yield the highest correla-
tions for relevance. The correlations with novelty and
experienced serendipity are also moderately high. Also
here, there does not seem to be much correlation with
diversity. This suggests that people prefer to obtain rec-
ommendations that are, above all, relevant, even though
novelty may also increase satisfaction. Further analysis
will reveal whether these effects can be explained by
experienced serendipity. The results, however, do not
suggest that users seek much diversity in their recom-
mendations in order to be satisfied.
When considering user conversion, lastly, the data
show slightly different patterns. Here, relevance does not
seem to correlate much higher than novelty, even though
all correlations are still moderately high. Also here, diver-
sity does not seem to affect conversion at all in the Activi-
ties scenario, but it does show a moderate correlation
with user conversion in the Catering scenario.
It should be noted that the correlation patterns are
very similar between the two experimental groups, that
is, in the context of recommender systems for catering
stores compared to recommender systems for urban
activities. This suggests that the context of recommender
system usage is of minor importance within these two
scenarios.
Before the models were fit, we also tested for
multicollinearity among the different items, because this
may result in unreliable parameter estimates (Farrar &
Glauber, 1967). Nonetheless, as none of the correlation
coefficients exceeds .80, there is a low risk of
multicollinearity.
4.2 |Mediation analysis
Considering the results of the path analysis (see Table 2
and Figure 2), the findings from the correlation matrix
are confirmed. The results suggest that experienced ser-
endipity largely depends on the relevance and the novelty
of the recommendations, even after controlling for the
other antecedents. Diversity in the recommendations, in
contrast, does not seem to affect experienced serendipity.
In the Activities group, the estimated effect of diversity
on experienced serendipity was even slightly negative
and statistically quite significant. All combined, the ante-
cedents explain almost 40% of the variance in experi-
enced serendipity in both the Catering and the Activities
group, which is moderately high in social sciences.
6SMETS ET AL.
TABLE 2 Relevance, novelty and serendipity do not seem to have an indirect effect on recommender system conversion through
satisfaction, but they do have a direct effect
Stand. effects
on serendipity
Standardized effects on Satisfaction Standardized effects on Conversion
Direct Indirect Total Direct Indirect Total
Catering group
Relevance .477
[<.001]
.444
[< .001]
.063
[.024]
.507
[<.001]
.308
[< .001]
.003
[.914]
.311
[<.001]
Novelty .252
[<.001]
.168
[.004]
.033
[.068]
.201
[<.001]
.127
[.019]
.001
[.913]
.128
[.037]
Diversity !.043
[.248]
!.029
[491]
!.006
[.233]
!.035
[.407]
.159
[<.001]
!.000
[.909]
.159
[<.001]
Serendipity .131
[.030]
.226
[<.001]
.001
[.914]
.227
[<.001]
Satisfaction .007
[.914]
R
2
.388 .390 .339
Activities group
Relevance .410
[<.001]
.526
[<.001]
.054
[.002]
.580
[<.001]
.235
[<.001]
!.003
[.917]
.232
[<.001]
Novelty .308
[<.001]
.095
[.007]
.040
[002]
.135
[<.001]
.125
[.001]
!.001
[.917]
.124
[.001]
Diversity !.116
[.001]
!.020
[.541]
!.015
[.014]
!.035
[.270]
.032
[.342]
.000
[.915]
.032
[.333]
Serendipity .131
[.001]
.394
[< .001]
!.001
[.917]
.393
[<.001]
Satisfaction !.005
[.917]
R
2
.390 .433 .388
Note: p-values between square brackets.
FIGURE 2 Our data suggest an
effect of relevance, novelty and
serendipity on recommender system
satisfaction and conversion. Satisfaction
does not seem to have an effect on
conversion
SMETS ET AL.7
When considering the standardized effects on user
satisfaction, relevance clearly shows the largest effect
(and for the Catering and Activities group respectively).
The more recommendations are relevant to the user, the
more satisfied this user will be. The effect of novelty of
the recommendations, in turn, is three to four times
smaller (and respectively), although these effects are still
highly significant. Similar to the effects on experienced
serendipity, the effect of diversity on user satisfaction is
almost non-existent in both experimental groups.
More surprisingly, however, the data do not confirm
that the effect of relevance and novelty on user satisfac-
tion can be explained by experienced serendipity. In both
experimental groups, the indirect effects of relevance and
novelty on satisfaction through experienced serendipity
are very small. Put differently, our results do not confirm
that higher user satisfaction due to high relevance and
novelty in recommendations can be explained by positive
experiences of serendipity. Also, after controlling for rele-
vance, novelty and diversity, the effect of experienced ser-
endipity on user satisfaction is much smaller compared
to the bivariate correlations (see Table 1). This suggests
that observed relations between experienced serendipity
and user satisfaction are spurious correlations because
they are both commonly caused by the relevance and
novelty of recommendations.
Looking at the effects on user conversion, also here,
the total effects of relevance and novelty remained fairly
large and statistically very significant in both the Cat-
ering and Activities scenarios, even after controlling for
the other variables including experienced serendipity and
satisfaction. This firstly suggests that the effects of rele-
vance and novelty on conversion can neither be
explained by better experiences of serendipity nor by
higher satisfaction with the recommended items.
Again, relevance of recommended items seems to
have a larger effect on user conversion compared to the
novelty of recommended items in both the Catering and
Activities group (.308 versus .127 and .235 versus .125,
respectively). Surprisingly, within the Catering group, the
results also yielded a moderate effect of .159 for diversity.
The more diverse recommended restaurants and bars are,
the more the respondents state to follow up on these rec-
ommendations. Within the Activities group, in contrast,
no such effect was found, similar to previous findings for
diversity.
In contrast to the effects on user satisfaction, the
results also yielded fairly large effects of experienced ser-
endipity on user conversion. In the Activities scenario,
the effect of experienced serendipity was even about
twice as large as any other effect (.393 more specifically).
These results imply that recommendations pleasantly
surprising users increase the chance of users following up
on these recommendations, and this can only marginally
be explained by the relevance and the novelty of the
recommendations.
The differences between the effects on satisfaction
and user conversion seem to be explainable by a surpris-
ing lack of correlation between satisfaction and user con-
version. Indeed, after controlling for relevance, novelty,
diversity and experienced serendipity, the effect of satis-
faction on user conversion completely disappears (that is
.007 and !.005 respectively). Given the correlation
between both constructs found in the bivariate analyses,
these results suggest that satisfaction only relates to user
conversion because both concepts are determined by rele-
vance, novelty and experienced serendipity. In contrasts
to our expectations, however, satisfaction does not seem
to have an important influence on (self-reported) user
conversion.
Last, it should be noted that no large differences were
found between the Catering and Activities scenarios,
except for the few effects already mentioned above. The
purpose of the urban recommender system does not seem
to have an important influence on the processes behind
experienced serendipity in our examples.
5|DISCUSSION
5.1 |User evaluations of serendipity in
urban recommender systems
This paper set out to study how users experience seren-
dipity in urban recommender systems and which charac-
teristics of the recommendations (novelty, relevance and
diversity) contribute to this. Previous work had already
studied experienced serendipity in recommender systems
in various domains and identified some antecedents
(Chen et al., 2019; Lutz et al., 2017), but few did this by a
large-scale evaluation in the particular urban context. As
a result, this paper presented a first exploration of the
topic but also opened avenues for further research, which
aligns with the relatively novelty of the study of serendip-
ity in urban recommender systems.
Overall, our findings about the antecedents contribut-
ing to experienced serendipity in urban recommender
systems are in line with findings from research in other
application domains. The results of our research provide
a clear affirmative answer to our first research question
(RQ1): within urban recommender systems for catering
stores and activities, experienced serendipity does primar-
ily depend on the relevance of the recommended items
and secondarily on the novelty of these items. Increases
in diversity among the recommended items, however, do
not seem to affect experienced serendipity. This is in line
8SMETS ET AL.
with other studies that also reported smaller effects for
diversity (Chen et al., 2019).
However, when considering our second (RQ2) and
third research question (RQ3) related to respectively user
satisfaction and conversion, our results provided more
complex conclusions. Firstly, the results of our survey did
confirm that relevance and novelty positively affect user
satisfaction with recommended items as well as the chance
that users follow up on these recommendations, that is,
user conversion. For diversity, in contrast, such an effect
was again completely absent. People thus get most satisfac-
tion from recommendations when the recommendations
are relevant and novel to them, while they barely care
about any diversity in these recommendations.
Secondly, however, the effects of relevance and nov-
elty on satisfaction and conversion can barely be
explained by increases in experienced serendipity. More-
over, next to relevance and novelty, experienced seren-
dipity also seems to have a separate effect on satisfaction
and user conversion. These results might suggest that
experienced serendipity should be considered as a con-
struct that acts next to the experienced relevance and
novelty of items in recommender systems. Future
research may focus on the distinction and relations
between all these concepts.
In addition, our results also revealed that user satisfac-
tion and user conversion are only spuriously related because
they commonly depend on factors like relevance, novelty,
diversity and experienced serendipity of recommended
items. After controlling for these factors, satisfaction and
conversion surprisingly do not seem to be related at all. This
unlikely lack of a direct relationship was, however, also
found in previous studies (Lutz et al., 2017).
Finally, for the contextual differentiation under study
in the fourth research question (RQ4), the results did not
show large differences between recommender systems for
catering stores and for urban activities. Put differently, the
context of the recommendations does not seem to have an
impact on the way serendipity related concepts determine
user satisfaction and conversion. One notable exception
was the effect of diversity on user conversion, which was
moderately large in the Catering scenario and non-existent
in the Activities scenario. It should be noted here, how-
ever, that the difference between recommender systems
for catering stores and urban activities might not suffi-
ciently reflect the diversity in urban recommender sys-
tems. Future research might thus elaborate on this topic.
5.2 |Limitations
We are aware that our research may have some methodo-
logical limitations. The first limitation is the use of a
retrospective survey about respondents' past experiences
and behavior. Retrospectively asking people for their
experiences may introduce measurement variance and
bias because reported experiences may not correspond
with true experiences at the moment the recommender
systems were used due to recall bias. As already discussed
above, a solution to this problem can be found in experi-
mental studies. Nonetheless, such studies, while optimiz-
ing internal quality, may suffer from lower external
quality because of the artificial settings they create. As a
result, in order to get full insights in the topic of serendip-
ity in urban recommender systems both experimental as
well as observational studies are required in order to tri-
angulate findings. For that reason, we believe our study
provides a contribution to the field.
Another alternative to solve the problem of recall bias
is the use of experience sampling, in which users are
immediately asked some questions in a pop-up directly
after using a recommender system. Experience sampling,
however, is very difficult to implement as it requires
adapting the recommender system user interface. Addi-
tionally, it may also annoy users leading to dropouts and
it put some serious restrictions on the number of ques-
tions that can be asked.
The second limitation is that no causal relationships
between the different concepts, as implied by the theoret-
ical model (see Figure 1), could be proven by the study
design. Such causal claims can only be investigated by
true experimental designs in which system design charac-
teristics related to relevance, novelty and diversity are
explicitly manipulated by the study experimenter. Unfor-
tunately, such experiments are still scarce in the existing
literature. Moreover, such experiments also do not allow
to make causal claims about the relationships between
the true experiences of relevance, novelty, diversity and
serendipity.
The third limitation is the fact that all concepts were
measured by single questions instead of item batteries.
Although this approach was also adopted in related work
Chen et al. (2019) this may lead to measurement bias.
Also, order effects may have played a role: the question
about user satisfaction was asked as the first question
while the conversion question as the last one. This might
also explain the lack of correlation between satisfaction
and conversion. Future research may focus on such order
effects, for example, by randomizing the question
order in order to neutralize such ordering effects. Fur-
ther, question wording may also lead to measurement
bias and variance due to interpretation differences among
respondents. For example, the question measuring diver-
sity may also be interpreted as how often the service
updates recommendations for users. Likewise, novelty
can be interpreted in different ways (Iaquinta et al., 2008;
SMETS ET AL.9
Kotkov, Wang, & Veijalainen, 2016) and it remains unclear
to what extend the interpretation overlaps between the
questions measuring novelty in particular and experienced
serendipity in general. Unfortunately, research on proper
question development about serendipity is still scarce and
very heterogeneous in the current literature and may thus
form an interesting topic for future research. Such research
will not only entail questionnaire development and valida-
tion but will also require more in-depth research on the
meaning of serendipity in urban environments (and urban
recommender systems in particular).
Further, in order to circumvent an artificial research
environment with a limited focusoncertainantecedentsof
serendipity, we asked our respondents to think of the rec-
ommender systems they would actually use in their general
daily life to find catering storesoractivitieswithintheircit-
ies instead of creating lab experiments to test for particular
design differences in recommender engines. However, as a
result of this strategy, we also do not know about which rec-
ommender systems our respondents thought about while
completing the survey and to what extent their momentary
reactions represent their overall opinions adequately.
Finally, our research strategy also depends on the cur-
rent status of existing urban recommender systems. This
means that our research merely involves an evaluation by
citizens of these current systems rather than an investiga-
tion of how citizens actually want such systems to be. For
example, it might well be the case that today's urban rec-
ommender systems lack a sufficient amount of diversity in
their recommendations, which make questions about such
diversity much more abstract to respondents.
A final remark should be made about the particular
timing of our survey, which was during the early days of
the global pandemic (March–April 2020). In Belgium, cit-
izens were since mid-March restricted in their move-
ments and so-called non-essential shops (including bars
and restaurants) had to close. Despite this situation, we
believe that it did not significantly impact our findings
since the survey retrospectively questioned citizens about
their past experiences and behavior. Since we asked these
questions just at the beginning of the pandemic, citizens
could still recall recent experiences. Nevertheless, if the
timing were to have an impact, it could potentially
explain the weak correlation between satisfaction and
conversion. However, further work here is needed.
6|CONCLUSION
This work aimed to contribute to the existing work on ser-
endipity by providing a first insight into user evaluations of
serendipity in urban recommender systems. By means of a
survey among 1,641 citizens in Flanders and Brussels
(Belgium) we collected data on their previous experiences
with using recommender systems in urban contexts. More
specifically, we explored to what extent characteristics of
the recommended items (i.e., their relevance, novelty and
diversity) led to experiences of serendipity and how this
relates to user satisfaction and conversion.
Our findings showed that users' experiences of seren-
dipity in urban recommender systems align with findings
in other application domains in the sense that there is a
strong relation between relevance, novelty and experi-
enced serendipity. Moreover, serendipitous recommenda-
tions are found to increase the chance of users following
up on these recommendations. A noteworthy finding is
the fact that diversity is only weakly correlated with expe-
rienced serendipity, similar to findings in other work. We
believe this result has to be interpreted carefully as it is
exactly the assumed lack of diversity that spurs research
into serendipitous recommendations. In other words, the
lack of diversity in today's recommender systems might
possibly explain this weak relationship.
By elaborating on the limitations of our study, we
underlined the difficulty of collecting data on user evalu-
ations of serendipity in urban recommender systems. We
therefore call for further research that studies this subject
in more depth, taking into account the previously
suggested paths for further work. Such research will con-
tribute to the current challenges that come along with
the increasing implementation of technologies in our
urban environments, and how this affects serendipity in
the city.
ORCID
Annelien Smets https://orcid.org/0000-0003-4771-7159
Jorre Vannieuwenhuyze https://orcid.org/0000-0002-
9820-7653
Pieter Ballon https://orcid.org/0000-0001-6066-3242
REFERENCES
Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise
up against statistical significance. Nature,567, 305–307.
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator vari-
able distinction in social psychological research: Conceptual,
strategic, and statistical considerations. Journal of Personality
and Social Psychology,51(6), 1173–1182.
Björneborn, L. (2017). Three key affordances for serendipity. Jour-
nal of Documentation,73, 1053–1081.
Chen, L., Yang, Y., Wang, N., Yang, K., & Yuan, Q. (2019). How
serendipity improves user satisfaction with recommendations?
A large-scale user evaluation. In The World Wide Web Confer-
ence (pp. 240–250).
De Gemmis, M., Lops, P., Semeraro, G., & Musto, C. (2015). An
investigation on the serendipity problem in recommender sys-
tems. Information Processing & Management,51(5), 695–717.
de Waal, M. (2013). De stad als interface: Hoe nieuwe media de stad
veranderen. Nai010.
10 SMETS ET AL.
Delva, H., Smets, A., Colpaert, P., Ballon, P., & Verborgh, R. (2020).
Using regions of interest for personalized route planning. In
Proceedings of the 2nd International Workshop on Semantics for
Transport.
Ekstrand, M. D., Harper, F. M., Willemsen, M. C., & Konstan, J. A.
(2014). User perception of differences in recommender algo-
rithms. In Proceedings of the 8th ACM Conference on Recom-
mender Systems (pp. 161–168).
Erdelez, S. (2004). Investigation of information encountering in the
controlled research environment. Information Processing &
Management,40(6), 1013–1025.
Erdelez, S., & Makri, S. (2020). Information encountering re-
encountered. Journal of Documentation,76, 731–751.
Farrar, D. E., & Glauber, R. R. (1967). Multicollinearity in regres-
sion analysis: The problem revisited. The Review of Economics
and Statistics,49(1), 92–107 Retrieved from http://www.jstor.
org/stable/1937887
Feise, R. J. (2002). Do multiple outcome measures require p-value
adjustment? BMC Medical Research Methodology,2(1), 1–4.
Foth, M. (2017). The software-sorted city: Big data & algorithms. In
Digital cities workshop. ACM.
Ge, M., Delgado-Battenfeld, C., & Jannach, D. (2010). Beyond accu-
racy: Evaluating recommender systems by coverage and seren-
dipity. In Proceedings of the Fourth ACM Conference on
Recommender Systems (pp. 257–260).
Ge, X., Daphalapurkar, A., Shimpi, M., Kohli, D., Pelechrinis, K.,
Chrysanthis, P. K., & Zeinalipour-Yazti, D. (2017). Data-driven
serendipity navigation in urban places. In 2017 IEEE 37th Inter-
national Conference on Distributed Computing Systems (ICDCS)
(pp. 2501–2504).
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C.,
Goodman, S. N., & Altman, D. G. (2016). Statistical tests,
pvalues, confidence intervals, and power: A guide to misinter-
pretations. European Journal of Epidemiology,31(4), 337–350.
Iaquinta, L., De Gemmis, M., Lops, P., Semeraro, G.,
Filannino, M., & Molino, P. (2008). Introducing serendipity in a
content-based recommender system. In 2008 Eighth Interna-
tional Conference on Hybrid Intelligent Systems (pp. 168–173).
Jacobs, J. (1961). The death and life of great American cities. Vintage.
Jannach, D., & Adomavicius, G. (2016). Recommendations with a
purpose. In Proceedings of the 10th ACM Conference on Recom-
mender Systems (pp. 7–10).
Jiang, T., Liu, F., & Chi, Y. (2015). Online information encounter-
ing: Modeling the process and influencing factors. Journal of
Documentation.,71, 1135–1157.
Kaminskas, M., & Bridge, D. (2016). Diversity, serendipity, novelty,
and coverage: A survey and empirical analysis of beyond-
accuracy objectives in recommender systems. ACM Transac-
tions on Interactive Intelligent Systems (TIIS),7(1), 1–42.
Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., &
Newell, C. (2012). Explaining the user experience of recom-
mender systems. User Modeling and User-Adapted Interaction,
22(4), 441–504.
Kotkov, D., Konstan, J. A., Zhao, Q., & Veijalainen, J. (2018). Inves-
tigating serendipity in recommender systems based on real user
feedback. In Proceedings of the 33rd Annual ACM Symposium
on Applied Computing (pp. 1341–1350).
Kotkov, D., Veijalainen, J., & Wang, S. (2016). Challenges of seren-
dipity in recommender systems. In Webist 2016: Proceedings of
the 12th International Conference on Web Information Systems
and Technologies. (Vol. 2).
Kotkov, D., Wang, S., & Veijalainen, J. (2016). A survey of serendip-
ity in recommender systems. Knowledge-Based Systems,111,
180–192.
Kunaver, M., & Požrl, T. (2017). Diversity in recommender
systems–a survey. Knowledge-Based Systems,123, 154–162.
Li, P., & Tuzhilin, A. (2019). Latent modeling of unexpectedness for
recommendations. In ACM recsys (late-breaking results)
(pp. 36–40). Copenhagen, Denmark.
Lutz, C., Pieter Hoffmann, C., & Meckel, M. (2017). Online seren-
dipity: A contextual differentiation of antecedents and out-
comes. Journal of the Association for Information Science and
Technology,68(7), 1698–1710.
MacKinnon, D. P. (2008). Introduction to statistical mediation analy-
sis. Taylor & Francis Group.
Makri, S., & Blandford, A. (2012). Coming across information
serendipitously—Part 1. Journal of Documentation,68(5), 684–705.
Maksai, A., Garcin, F., & Faltings, B. (2015). Predicting online per-
formance of news recommender systems through richer evalua-
tion metrics. In Proceedings of the 9th ACM Conference on
Recommender Systems (pp. 179–186).
Matt, C., Benlian, A., Hess, T., & Weiß, C. (2014). Escaping from
the filter bubble? The effects of novelty and serendipity on
users' evaluations of online recommendations. In Icis2014
proceedings.
McCay-Peet, L. (2014). Investigating work-related serendipity, what
influences it, and how it may be facilitated in digital environ-
ments (Unpublished doctoral dissertation).
McCay-Peet, L., & Toms, E. (2011). Measuring the dimensions of
serendipity in digital environments. Information Research: An
International Electronic Journal,16(3), n3.
McQuire, S. (2017). Geomedia: Networked cities and the future of
public space. John Wiley & Sons.
Olshannikova, E., Olsson, T., Huhtamäki, J., Paasovaara, S., &
Kärkkäinen, H. (2020). From chance to serendipity: Knowledge
workers' experiences of serendipitous social encounters.
Advances in Human-Computer Interaction,2020,1–18.
Pariser, E. (2011). The filter bubble: What the internet is hiding from
you. Penguin.
Paulos, E., & Goodman, E. (2004). The familiar stranger: Anxiety,
comfort, and play in public places. In Proceedings of the Sigchi
Conference on Human Factors in Computing Systems
(pp. 223–230).
Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation frame-
work for recommender systems. In Proceedings of the Fifth
ACM Conference on Recommender Systems (pp. 157–164).
Quijano-S!
anchez, L., Cantador, I., Cortés-Cediel, M. E., & Gil, O.
(2020). Recommender systems for smart cities. Information Sys-
tems,92(101545),
Reviglio, U. (2019). Towards a taxonomy for designing serendipity
in personalized news feeds. In Proceedings of the 10th Interna-
tional Conference on Conceptions of Library and Information
Science. University of Borås.
Rosseel, Y. (2012). lavaan: An r package for structural equation
modeling. Journal of Statistical Software, Articles,48(2), 1–36.
https://doi.org/10.18637/jss.v048.i02
Rothman, K. J. (1990). No adjustments are needed for multiple
comparisons. Epidemiology,1, 43–46.
SMETS ET AL.11
Rucker, D. D., Preacher, K. J., Tormala, Z. L., & Petty, R. E. (2011).
Mediation analysis in social psychology: Current practices and
new recommendations. Social and Personality Psychology Com-
pass,5(6), 359–371.
Said, A., Fields, B., Jain, B. J., & Albayrak, S. (2013). User-centric
evaluation of a k-furthest neighbor collaborative filtering rec-
ommender algorithm. In Proceedings of the 2013 Conference on
Computer Supported Cooperative Work (pp. 1399–1408).
Shepard, M. (2011). Sentient city: Ubiquitous computing, architec-
ture, and the future of urban space. The MIT press.
Silveira, T., Zhang, M., Lin, X., Liu, Y., & Ma, S. (2019). How good
your recommender system is? A survey on evaluations in rec-
ommendation. International Journal of Machine Learning and
Cybernetics,10(5), 813–831.
Smets, A., Montero, E., & Ballon, P. (2019). Does the bubble go
beyond? An exploration of the urban filter bubble. In The 1st
Workshop on the Impact of Recommender Systems with ACM
Recsys 2019. CEUR Workshop Proceedings.
Sun, X., Sharples, S., & Makri, S. (2011). A user-centred mobile
diary study approach to understanding serendipity in informa-
tion research. Information Research,16(3), 16–13.
Tacchini, E. (2012). Serendipitous mentorship in music recommender
systems (unpublished doctoral dissertation).
van der Graaf, S., & Ballon, P. (2019). Navigating platform urbanism.
Technological Forecasting and Social Change,142, 364–372.
Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and
use of information technology: Extending the unified theory of
acceptance and use of technology. MIS Quarterly,36(1), 157–178.
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-
values: Context, process, and purpose. The American Statisti-
cian,70(2), 129–133.
Wood, P., & Landry, C. (2008). The intercultural city: Planning for
diversity advantage. Routledge.
Zhang, Y. C., Séaghdha, D. Ó., Quercia, D., & Jambor, T. (2012).
Auralist: Introducing serendipity into music recommendation.
In Proceedings of the Fifth ACM International Conference on
Web Search and Data Mining (pp. 13–22).
Zuiderveen Borgesius, F., Trilling, D., Möller, J., Bod!
o, B., De
Vreese, C. H., & Helberger, N. (2016). Should we worry about
filter bubbles? Internet Policy Review. Journal on Internet Regu-
lation,5(1). https://doi.org/10.14763/2016.1.401.
How to cite this article: Smets, A.,
Vannieuwenhuyze, J., & Ballon, P. (2021).
Serendipity in the city: User evaluations of urban
recommender systems. Journal of the Association
for Information Science and Technology,1–12.
https://doi.org/10.1002/asi.24552
APPENDIX
Questions catering group
1. Searching for a bar or restaurant on the internet. How
do you experience this? How often do you use
websites or apps (e.g., search robots like Google,
TripAdvisor, …) to search for new restaurants or bars?
Very often—Often—Sometimes—Rarely—Never
2. How satisfied are you with the recommended restau-
rants or bars on these websites or apps? Very satis-
fied—Satisfied—Neither satisfied, nor dissatisfied—
Dissatisfied—Very dissatisfied
3. How often do you think such websites or apps recom-
mend restaurants or bars that suit you well? Very
often —Often —Sometimes —Rarely —Never
4. How often do you think such websites and apps rec-
ommend restaurants or bars you do not know yet?
Very often—Often—Sometimes—Rarely—Never
5. How often do you think you get the same kind of res-
taurants or bars recommended on these websites and
apps? Very often—Often—Sometimes—Rarely—Never
6. How often do you find yourself pleasantly surprised
by the recommended restaurants or bars on these
websites and apps? Very often—Often—Sometimes—
Rarely—Never
7. How often do you actually go to the recommended
restaurants or bars on these websites and apps? Very
often—Often—Sometimes—Rarely—Never
Questions activities group
1. Searching for activities on the internet. How do you
experience this? How often do you use websites or
apps (e.g., search robots such as Google, TripAdvisor,
…) to find out what to do in a city (e.g., when you are
on vacation or planning a day trip)? Very often—
Often—Sometimes—Rarely—Never
2. How satisfied are you with the recommended restau-
rants or bars on these websites or apps? Very satis-
fied—Satisfied—Neither satisfied, nor dissatisfied—
Dissatisfied—Very dissatisfied
3. How often do you think such websites or apps recom-
mend restaurants or bars that suit you well? Very
often—Often—Sometimes—Rarely—Never
4. How often do you think such websites and apps rec-
ommend activities you do not know yet? Very often—
Often—Sometimes—Rarely—Never
5. How often do you think you get the same kind of
activities recommended on these websites and apps?
Very often—Often—Sometimes—Rarely—Never
6. How often do you find yourself pleasantly surprised
by the recommended activities on these websites and
apps? Very often—Often—Sometimes—Rarely—
Never
7. How often do you actually go to or actually participate
in the recommended activities on these websites and
apps? Very often—Often—Sometimes—Rarely—
Never
12 SMETS ET AL.
