Active Learning in Recommender Systems

Abstract

In Recommender Systems (RS), a users preferences are expressed in
terms of rated items, where incorporating each rating may improve the RS’s predictive
accuracy. In addition to a user rating items at-will (a passive process), RSs
may also actively elicit the user to rate items, a process known as Active Learning
(AL). However, the number of interactions between the RS and the user is still limited.
One aim of AL is therefore the selection of items whose ratings are likely to
provide the most information about the user’s preferences. In this chapter, we provide
an overview of AL within RSs, discuss general objectives and considerations,
and then summarize a variety of methods commonly employed. AL methods are
categorized based on our interpretation of their primary motivation/goal, and then
sub-classified into two commonly classified types, instance-based and model-based,
for easier comprehension. We conclude the chapter by outlining ways in which AL
methods could be evaluated, and provide a brief summary of methods performance.

Full text

Chapter 24
Active Learning in Recommender Systems
Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
Abstract In Recommender Systems (RS), a users preferences are expressed in
terms of rated items, where incorporating each rating may improve the RS’s pre-
dictive accuracy. In addition to a user rating items at-will (a passive process), RSs
may also actively elicit the user to rate items, a process known as Active Learning
(AL). However, the number of interactions between the RS and the user is still lim-
ited. One aim of AL is therefore the selection of items whose ratings are likely to
provide the most information about the user’s preferences. In this chapter, we pro-
vide an overview of AL within RSs, discuss general objectives and considerations,
and then summarize a variety of methods commonly employed. AL methods are
categorized based on our interpretation of their primary motivation/goal, and then
sub-classified into two commonly classified types, instance-based and model-based,
for easier comprehension. We conclude the chapter by outlining ways in which AL
methods could be evaluated, and provide a brief summary of methods performance.
24.1 Introduction
Recommender Systems (RSs) are often assumed to present items to users for one
reason – to recommend items a user will likely be interested in. However, there
is another reason for presenting items to users: to learn more about their prefer-
Neil Rubens
University of Electro-Communications, Tokyo, Japan, e-mail: rubens@hrstc.org
Mehdi Elahi
Free University of Bozen-Bolzano e-mail: mehdi.elahi@unibz.it
Masashi Sugiyama
Tokyo Institute of Technology, Tokyo, Japan e-mail: sugi@cs.titech.ac.jp
Dain Kaplan
Tokyo Institute of Technology, Tokyo, Japan e-mail: dain@cl.cs.titech.ac.jp
819

820 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
ences. This is where Active Learning (AL) comes in. Augmenting RSs with AL
helps the user become more self-aware of their own likes/dislikes while at the same
time providing new information to the system that it can analyze for subsequent
recommendations. In essence, applying AL to RSs allows for personalization of the
recommending process, a concept that makes sense as recommending is inherently
geared towards personalization. This is accomplished by letting the system actively
influence which items the user is exposed to (e.g. the items displayed to the user
during sign-up or during regular use), and letting the user explore his/her interests
freely.
Unfortunately, there are very few opportunities for the system to acquire infor-
mation, such as when a user rates/reviews an item, or through a user’s browsing
history. Since these opportunities are few, we want to be as sure as possible that the
data we acquire tells us something important about the user’s preferences. After all,
one of the most valuable assets of a company is user data.
For example, when a new user starts using a recommender system, very little is
known about his/her preferences [55, 47, 2]. A common approach to learning the
user’s preferences is to ask him/her to rate a number of items (known as training
points). A model that approximates their preferences is then constructed from this
data. Since the number of items reviewed by the user cannot span the system’s entire
catalog (and indeed would make the task of AL as well as recommending moot
points), the collection of items presented to the user for review must necessarily
be very limited. The accuracy of the learned model thus greatly depends on the
selection of good training points. A system might ask the user to rate Star Wars
I,II, and III. By rating all three volumes of this trilogy, we will have a good idea
of the user’s preferences for Star Wars, and maybe by extension, an inclination for
other movies within the Sci-Fi genre, but overall the collected knowledge will be
limited. It is therefore unlikely that picking the three volumes of a trilogy will be
informative.1Another issue with selecting a popular item such as Star Wars is that
by definition the majority of people like them (or they would not be popular). It is
not surprising then, that often little insight is gained by selecting popular items to
learn about the user (unless the user’s tastes are atypical).
There is a notion that AL is a bothersome, intrusive process, but it does not have
to be this way [66, 50]. If the items presented to the user are interesting, it could
be both a process of discovery and of exploration. Some Recommender Systems
provide a “surprise me!” button to motivate the user into this explorative process,
and indeed there are users who browse suggestions just to see what there is without
any intention of buying. Exploration is crucial for users to become more self-aware
of their own preferences (changing or not) and at the same time inform the system
of what they are. Keep in mind that in a sense users can also be defined by the items
they consume, not only by the ratings of their items, so by prompting users to rate
different items it may be possible to further distinguish their preferences from one
another and enable the system to provide better personalization and to better suit
their needs.
1Unless our goal is to learn a kind of micro-preference, which we can define as a person’s tendency
to be more ’picky’ concerning alternatives close to one another in an genre they like.

24 Active Learning in Recommender Systems 821
This chapter is only a brief foray into Active Learning in Recommender Systems.
2We hope that this chapter can, however, provide the necessary foundations.
For further reading, [57] gives a good, general overview of AL in the context
of Machine Learning (with a focus on Natural Language Processing and Bioinfor-
matics). For a theoretical perspective related to AL (a major focus in the field of
Experimental Design), see [7, 4, 28]; there have also been recent works in Com-
puter Science [17, 5, 62].
24.1.1 Objectives of Active Learning in Recommender Systems
Different RSs have different objectives (Chapter 8), which necessitate different ob-
jectives for their Active Learning components as well. As a result, one AL method
may be better suited than another for satisfying a given task [46]. For example, what
is important in the recommender system being built (Chapter 9)? The difficulty of
signing-up (user effort)? If the user is happy with the service (user satisfaction)?
How well the system can predict a user’s preferences (accuracy)? How well the sys-
tem can express a user’s preferences (user utility)? How well the system can serve
other users by what it learns from this one (system utility)? System functionality
may also be important, such as when a user inquires about a rating for an item of
interest the system has insufficient data to predict a rating for, what the system does
in response. Does it in such a case give an ambiguous answer, allowing the user to
train the system further if they have the interest and the time to do so? Or does it
require them to rate several other items before providing a prediction? Perhaps the
user has experienced the item (e.g. watched the movie or trailer) and thinks their
rating differs substantially from the predicted one [11]. In all these cases how the
system responds to the user is important for consideration.
Traditionally AL does not consider the trade-off of exploration (learning user’s
preferences) and exploitation (utilizing user’s preferences), that is, it does not dy-
namically assign weights to exploitation/exploration depending on system objec-
tives. This trade-off is important because for a new user about which nothing or
little is known, it may be beneficial to validate the worth of the system by providing
predictions the user is likely to be interested in (exploitation), while long-term users
may wish to expand their interests through exploration [50, 52].
Though an objective of the RS will likely be to provide accurate predictions to
the user, the system may also need to recommend items of high novelty/serendipity
26, improve coverage, maximize profitability, or determine if the user is even able to
evaluate a given item, to name a few [55, 27, 43]. Multiple objectives may need to be
considered simultaneously (Chapter 25), e.g. minimizing the net acquisition cost of
training data while maximizing net profit, or finding the best match between the cost
of offering an item to the user, the utility associated with expected output, and the
alternative utility of inaction [50]. The utility of training may also be important, e.g.
2Supplementary materials on Active Learning can be found at: http://
ActiveIntelligence.org

822 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
predicting ratings for exotic cars may not be so useful if the user is not capable of
purchasing them and so should be avoided. It can be seen that the system objective
is often much more complex than mere predictive accuracy, and may include the
combination of several objectives.
While Recommender Systems often have an ill-defined or open-ended objective,
namely to predict items a user would be ‘interested’ in, Conversation-based AL
[42, 49, 9], as the name suggests, engages in a conversation with the user as a goal
oriented approach. It seeks to, through each iteration of questioning, elicit a response
from the user to best reduce the search space for quickly finding what it is the user
seeks (see Section 24.7).
The New User Problem When a user starts using a RS they expect to see in-
teresting results after a minimal amount of training. Though the system knows
little about their preferences, it is essential that training points are selected for
rating by the user that will maximize understanding what the new user wants
[46].
The New Product Problem As new products are introduced into the sys-
tem, it is important to quickly improve prediction accuracy for these items by
selecting users to rate them [30].
Cost of obtaining an output value Different means of obtaining an output
value come at different costs. Implicit strategies, such as treating a user click
on a suggested item as positive output, or not clicking as negative, are inex-
pensive in relation to user effort. Conversely, asking the user to explicitly rate
an item is more costly, though still dependent on the task. Watching a movie
like Star Wars to rate may provide good results but requires substantial user
effort [25]; rating a joke requires much less. This often dovetails the explo-
ration/exploitation coupling and trade-offs between obtaining outputs from
different inputs should also be considered (e.g. certainty/uncertainty, ease of
evaluation, etc.)
Adaptation for different AL methods Though we focus on the traditional
objective of reducing predictive error, it is equally plausible to construct a
method for maximizing other goals, such as profitability. In this case a model
would pick points that most likely increase profit rather than a rating’s accu-
racy.
24.1.2 An Illustrative Example
Let’s look at a concrete example of Active Learning in a Recommender System.
This is only meant to demonstrate concepts, so it is oversimplified. Please note that

24 Active Learning in Recommender Systems 823
c
a
b
input1
input2
c
a
b
input1
input2
test point (unrated)
training point
(color may differ)
ratings color map
41 32 5
Actual Ratings (Unknown)
dd
Fig. 24.1: Active Learning: illustrative example (See Section 24.1.2).
the similarity metric may differ depending on the method used; here, movies are
assumed to be close to one another if they belong to the same genre. Figure 24.1
on page 823 shows two charts, the leftmost is our starting state, in which we have
already asked the user to rate a movie within the upper right group, which we will
say is the Sci-Fi genre. The right chart shows us four possibilities for selecting our
next training point: (a), (b), (c), or (d). If we select the training point (a) which is an
obscure movie (like The Goldfish Hunter), it does not affect our predictions because
no other movies (points) are nearby. If we select the training point (b), we can predict
the values for the points in the same area, but these predictions are already possible
from the training point in the same area (refer to the chart on the left). If training
point (c) is selected, we are able to make new predictions, but only for the other
three points in this area, which happens to be Zombie movies. By selecting training
point (d), we are able to make predictions for a large number of test points that are
in the same area, which belong to Comedy movies. Thus selecting (d) is the ideal
choice because it allows us to improve accuracy of predictions the most (for the
highest number of training points).3
24.1.3 Types of Active Learning
AL methods presented in this chapter have been categorized based on our interpre-
tation of their primary motivation/goal. It is important to note, however, that various
ways of classification may exist for a given method, e.g. sampling close to a deci-
sion boundary may be considered as Output Uncertainty-based since the outputs are
unknown, Parameter-based because the point will alter the model, or even Decision
boundary-based because the boundary lines will shift as a result. However, since the
sampling is performed with regard to decision boundaries, we would consider this
the primary motivation of this method and classify it as such.
3This may be dependent on the specific prediction method used in the RS.

824 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
In addition to our categorization by primary motivation (Section 24.1), we further
subclassify a method’s algorithms into two commonly classified types for easier
comprehension: instance-based and model-based.
Instance-based Methods A method of this type selects points based on their prop-
erties in an attempt to predict the user’s ratings by finding the closest match to other
users in the system, without explicit knowledge of the underlying model. Other
common names for this type include memory-based, lazy learning, case-based, and
non-parametric [2].
Model-based Methods A method of this type selects points in an attempt to best
construct a model that explains data supplied by the user to predict user ratings [2].
These points are also selected to maximize the reduction of expected error of the
model. Model-based AL methods often achieve better performance than instance-
based methods, since they utilize not only the properties of the points (as instance-
based methods do), but also are able to optimize label acquisition with regards to
the underlying predictive model. However, the improved performance comes at a
cost. The labeled data obtained for one predictive model may not be as useful for
another; and predictive models do frequently change throughout the lifecycle of the
RS. Moreover, an AL method designed for one model is often incompatible with a
different model since model-based AL methods rely on access to specific parame-
ters of a model and the model’s estimates (e.g. not all of the models are able to pro-
vide a distribution of rating estimates, and models are parameterized differently);
this might necessitate using a different AL method each time the predictive model
changes.
Modes of Active Learning: Batch and Sequential Because users typically
want to see the system output something interesting immediately, a common
approach is to recompute a user’s predicted ratings after they have rated a
single item, in a sequential manner. It is also possible, however, to allow a
user to rate several items, or several features of an item before readjusting
the model. On the other hand, selecting training points sequentially has the
advantage of allowing the system to react to the data provided by users and
make necessary adjustments immediately. Though this comes at the cost of
interaction with the user at each step. Thus a trade-off exists between Batch
and Sequential AL: the usefulness of the data vs. the number of interactions
with the user.
24.2 Properties of Data Points
When considering any Active Learning method, the following three factors should
always be considered in order to maximize the effectiveness of a given point. Sup-

24 Active Learning in Recommender Systems 825
plementary explanations are then given below for the first two. Examples refer to
the Illustrative Example (Figure 24.1 on page 823).
(R1) Represented: Is it already represented by the existing training set? E.g.
point (b).
(R2) Representative: Is the point a good candidate for representing other data
points? Or is it an outlier? E.g. point (a).
(R3) Results: Will selecting this point result in better prediction ratings or
accomplish another objective? E.g. point (d), or even point (c).
(R1) Represented by the Training Data As explained in the introduction to this
chapter, asking for ratings of multiple volumes from a trilogy, such as Star Wars,
is likely not beneficial, as it may not substantially contribute to the acquisition of
new information about the user’s preferences. To avoid obtaining redundant infor-
mation, therefore an active learning method should favor items that are not yet well
represented by the training set [23].
(R2) Representative of the Test Data It is important that any item selected for
being rated by an AL algorithm be as representative of the test items as possible
(we consider all items as potentially belonging to the test set), since the accuracy
of the algorithm will be evaluated based on these items. If a movie is selected from
a small genre, like Zombie movies from the Illustrative Example (Figure 24.1 on
page 823), then obtaining a rating for this movie likely provides little insight into
a user’s preferences other, more prominent genres. In addition, users naturally tend
to rate movies from genres they like, meaning that any genre that dominates the
training set (which is likely composed of items the user likes) may be representative
of only a small portion of all items [50]. In order to increase information obtained,
it is important to select representative items which may provide information about
the other yet unrated items [23, 58, 64].
(R3) Results Active learning methods are typically evaluated based on how well
they assisted a recommender system in achieving its objectives (e.g. accuracy, cov-
erage, precision, etc. (Chapter 8). A common objective of RSs is high predictive
accuracy; hence active learning methods are primarily evaluated based on the same
metric. There are also some AL-centric metrics. A common AL-centric metric re-
flects the number of acquired ratings. In addition to measuring the quantity of the
elicited ratings, it is also important to measure the type of elicited ratings (e.g. rat-
ings low/high, genre, etc.) [20]. Many of the objectives have also been adopted from
other fields, in particularly from the field of information retrieval: precision, cumu-
lative gain 4(a measure for ranking quality of an item based on its relevance and
position in the list (i.e. high rated items should appear towards the top of the rec-
ommendation list)) (Chapter 8). Finally, it is important to closely emulate the actual
settings in which results are obtained (Section 24.8).
4for comparing of recommendations with various lengths, normalized Discounted Cumulative
Gain (NDCG) is frequently used

826 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
24.2.1 Other Considerations
In addition to the three Rs listed in Section 24.2, it may also be desirable to consider
other criteria for data points, such as the following.
Cost As touched upon in the introduction to this chapter, obtaining implicit feed-
back from user selections is cheaper than asking the user to explicitly rate an item
[24]. This can be considered a variable cost problem. One approach for tackling this,
is to take into account both the cost of labeling an item and the future cost of esti-
mated misclassification were the item to be added to the training set [35]. Moreover,
the cost may be unknown beforehand [59].
Ratability A user may not always be able to provide a rating for an item; you
cannot properly rate a movie you have not seen! It is suggested therefore that the
probability of a user being able to evaluate an item also be considered (Section
24.8.4).
Saliency Decision-centric AL places emphasis on items whose ratings are more
likely to affect decision-making, and acquires instances that are related to decisions
for which a relatively small change in their estimation can change the order of top
rated predictions [54]. For example, unless labeling an item would result in displac-
ing or rearranging a list of top ten recommended movies on a user’s home page (the
salient items), it may be considered of little use. It is also possible to only consider
the effect of obtaining an item’s rating on items that are strongly recommended by
the system [6].
Popularity It has also been suggested to take an item’s popularity into account
[46], i.e. how many people have rated an item. This operates on the principle that
since a popular item is rated by many people, it may be rather informative. Con-
versely, an item’s rating uncertainty should also be considered since popular items
have a tendency to be rated highly by most users (the reason for it being popular),
indicating that the item may not provide much discriminative power and thus not
worth including in the training set. This limitation has been partially addressed in
[36] by selecting popular items (in a personalized manner) among similar users.
Best/Worst The ratings with extreme values (best/worst) are often quite informa-
tive about both the user’s and items’ preferences [39]. One way to utilize this is
to ask the user to rate items with the highest [20, 65] and lowest predicted ratings
[18, 20]. Note that the highest-predicted is the default strategy used by RSs to ac-
quire ratings. However, concentrating on obtaining only highly rated items could
introduce a system-wide bias [21] and could result in degradation of predictive per-
formance [20]. Highest-lowest strategy is more likely to present items that users are
able to rate (likely to have experienced the highest-predicted items, and can prob-
ably easily express a negative opinion about the lowest-predicted items). A major
drawback of this method is that the system tends to obtain new information when
its predictions are wrong (which we would hope is not so frequent). We hypothe-
size that this problem could be alleviated by asking a user to provide his/her most

24 Active Learning in Recommender Systems 827
liked/disliked items. However, this changes the type of the task from active learning
(providing a label for an item), to active class selection [40] (providing an item with
a certain label (liked/disliked)).
24.3 Active Learning in Recommender Systems
With traditional AL, users are asked to rate a set of preselected items. This is often at
the time of enrollment, though a preselected list may be presented to existing users
at a later date as well. It may be argued that since these items are selected by experts,
they capture essential properties for determining a user’s preferences. Conceptually
this may sound promising, but in practice this often leads towards selecting items
that best predict the preferences of only an average user. Since the idea of RS is
to provide personalized recommendations, selecting items to rate in a personalized
manner should readily make more sense. The following matrix (Table 24.1 on page
828) provides a summary of the methods overviewed in this chapter.
24.3.1 Active Learning Formulation
Passive Learning (see Figure 24.2 on page 827) refers to when training data is pro-
vided beforehand, or when the system makes no effort to acquire new data (it simply
accumulates through user activities over time). Active Learning, on the other hand,
selects training points actively (the input) so as to observe the most informative
output (user ratings, behavior, etc.).
Passive Learning
Active Learning
training data
approximated
function
supervised
learning
user
training data
request
Fig. 24.2: Active learning employs an interactive/iterative process for obtaining
training data, unlike passive learning, where the data is simply given.
Let us define the problem of active learning in a more formal manner. An item
is considered to be a multi-dimensional input variable and is denoted by a vector

828 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
Primary Motivation of
Approach Description/Goal Possible Considerations
Uncertainty Reduction
(Section 24.4) Reducing uncertainty of:
•rating estimates (Section
24.4.1),
•decision boundaries (24.4.2),
•model parameters (24.4.3).
Reducing uncertainty may
not always improve accu-
racy; the model could simply
be certain about the wrong
thing (e.g. when the predic-
tive method is wrong).
Error Reduction
(Section 24.5) Reducing the predictive error
by utilizing the relation be-
tween the error and:
•the changes in the output es-
timates (Section 24.5.1.1),
•the test set error (24.5.1.2),
•changes in parameter esti-
mates (24.5.2.1),
•the variance of the parameter
estimates (24.5.2.2).
Estimating reduction of error
reliably could be difficult and
computationally expensive.
Ensemble-based
(Section 24.6) Identifying useful training
points based on consensus
between:
•models in the ensemble (Sec-
tion 24.6.1),
•multiple candidate models
(Section 24.6.1).
The effectiveness depends on
the quality of models/candi-
dates, and could be compu-
tationally expensive since it
is performed with regards to
multiple models/candidates.
Table 24.1: Method Summary Matrix.
x(also referred to as a data point).5The set of all items is denoted by X. The
preferences of a user uare denoted by a function fu(also referred to as a target
function); for brevity, we use fwhen referring to a target user. A rating of an item x
is considered to be an output value (or label) and is denoted as y=f(x). Each item
xcould be rated on a finite scale Y={1,2,...,5}.
In supervised learning, the items and corresponding user ratings are often par-
titioned into complementary subsets – a training set and a testing set (also called
a validation set). The task of supervised learning is then too, given a training set
(often supplemented by the ratings of all users), learn a function b
fthat accurately
approximates a user’s preferences. Items that belong to the training set are denoted
by X(Train), and these items along with their corresponding ratings constitute a train-
ing set, i.e. T={(xi,yi)}xi∈X(Train). We measure how accurately the learned function
5The way in which an item is represented depends on the RS and the underlying predictive method.
In Collaborative Filtering based approaches items could represented through the ratings of the
users, or, in content based RSs, items could be represented through their descriptions.

24 Active Learning in Recommender Systems 829
predicts the true preferences of a user by the generalization error:
G(b
f) = ∑
x∈X
Lf(x),b
f(x)P(x).(24.1)
In practice, however, f(x)is not available for all x∈X; it is therefore common to
approximate the generalization error by the test error:
b
G(b
f) = ∑
x∈X(Test)
Lf(x),b
f(x)P(x),(24.2)
where X(Test)refers to the items in the test set, and prediction errors are measured
by utilizing a loss function L, e.g. mean absolute error (MAE):
LMAE f(x),b
f(x)=f(x)−b
f(x),(24.3)
or mean squared error (MSE):
LMSE f(x),b
f(x)=f(x)−b
f(x)2.(24.4)
The active learning criterion is defined so as to estimate the usefulness of ob-
taining a rating of an item xand adding it to the training set X(Train)for achieving
a certain objective (Section 24.1.1). For simplicity, let us consider this objective to
be the minimization of generalization error of a learned function with respect to the
training set. We then denote the active learning criterion as:
b
G(X(Train)∪{x}),(24.5)
or for brevity, denote it as: b
G(x).(24.6)
The goal of active learning is to select an item xthat would allow us to minimize
the generalization error b
G(x):
argminxb
G(x).(24.7)
If we consider asking a user to rate an item xjor an item xk, then we would
estimate their usefulness by an active learning criterion, i.e. b
G(xj)and b
G(xk), and
select the one that will result in a smaller generalization error. Note that we need
to estimate the usefulness of rating an item without knowing its actual rating. To
distinguish a candidate item to be rated from the other items we refer to it as xa.
AL can be applied to any predictive method as long as it provides the required
information, such as rating estimates [53] and their distribution [29, 31], closeness
to the decision boundary [67, 16], method parameters [60], etc.
Regression and Classification The problem of predicting a user’s ratings could be
treated as both a regression and a classification problem. It is a regression problem
since the ratings are discrete numerical values,such as if we consider their ordi-

830 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
xinput (item)
Xinputs (items)
youtput (item’s rating)
Y={1,2,...,5}possible outputs (ratings), i.e. y∈Y
fuser’s preferences function (unknown to the system)
X(Train)training inputs (rated items)
T={(xi,yi}xi∈X(Train)training set (items and their ratings)
b
fapproximated function of user’s preferences (from training set)
Ggeneralization error (predictive accuracy) see (24.1)
xaitem considered for rating
b
G(xa)active learning criterion (estimates the usefulness of an item xa)
Table 24.2: Summary of Notation.
nal properties, meaning the ratings could be ordered (e.g. a rating of 4 is higher
than a rating of 3). On the other hand, we can disregard the numerical properties
of the ratings and treat the problem as a classification one by treating ratings as
classes/labels.6For example, we can use a nearest-neighbor (NN) approach to do
classification, e.g. pick the most frequent label of the neighbors; or we can use NN
to do regression, e.g. calculate the mean of the ratings of the neighbors. Throughout
the chapter we use both classification and regression in examples, selecting the one
most appropriate for aiding the current explanation.
24.4 Uncertainty-based Active Learning
Uncertainty-based AL tries to obtain training points so as to reduce uncertainty in
some aspect, such as concerning output values [37], the model’s parameters [29], a
decision boundary [56], etc. A possible drawback to this approach is that reducing
uncertainty may not always be effective. If a system becomes certain about user
ratings, it does not necessarily mean that it will be accurate, since it could simply be
certain about the wrong thing (i.e., if the algorithm is wrong, reducing uncertainty
will not help). As an example, if the user has so far rated items positively, a system
may mistakenly be certain that a user likes all of the items, which is likely incorrect.
6If the ordinal properties of the labels are considered, it is referred to as Ordinal Classification.

24 Active Learning in Recommender Systems 831
24.4.1 Output Uncertainty
In Output Uncertainty-based methods, an item to label (training point) is selected
so as to reduce the uncertainty of rating predictions for test items. In Figure 24.1
on page 823, with the assumption that the RS estimates the rating of an item based
on the cluster to which it belongs (e.g. items in the same movie genre receive the
same rating), if a user’s rating for a movie from the Sci-Fi genre (upper-right) has
already been obtained, then there is a higher likelihood that the RS may be more
certain about the ratings of other movies in the Sci-Fi genre, likely making it more
beneficial to obtain a user’s preference for a movie from a genre (cluster) not yet
sampled, i.e. a cluster that is still uncertain.
The difference between instance-based and model-based approaches for Output
Uncertainty-based AL is primarily in how for an arbitrary item xthe rating’s distri-
bution P(Yx)is obtained, where a rating’s distribution is defined as the probability
of an item being assigned a certain rating. For model-based methods it is possible to
obtain the rating’s distribution from the model itself. Probabilistic models are partic-
ularly well suited for this as they directly provide the rating’s distribution [29, 31].
For instance-based methods, collected data is used to obtain the rating’s distribution.
As an example, methods utilizing nearest-neighbor techniques can obtain a rating’s
distribution based on the votes of its neighbors, where “neighbor” here means a user
with similar preferences,7using a formula such as:
P(Yx=y) = ∑nn∈N Nx,ywnn
∑nn∈NNxwnn
,(24.8)
where NNxare neighbors that have rated an item x, and NNx,yare neighbors that
have given an item xa rating of y, and wnn is the weight of the neighbor (such as
similarity).
24.4.1.1 Active Learning Methods
Some AL methods [37] estimate the usefulness of a potential training point in a
local (greedy) manner by measuring the uncertainty of its output value:
b
GUncertaintylocal (xa) = −Uncertainty(Ya).(24.9)
Since our goal is to minimize b
G, rating an item with high uncertainty is useful; it will
eliminate the uncertainty about the rating of the chosen item. However, labeling an
item whose rating is uncertain does not necessarily accomplish the goal of reducing
the uncertainty of ratings for other items (e.g. labeling an outlier may only reduce
rating uncertainty for a few other similar items, such as when selecting item (c) in
the Zombie genre, or even none as in (d), shown in Figure 24.1 on page 823.
7Defining a neighbor as a similar item is also feasible depending on the method.

832 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
We may thus consider reducing uncertainty in a global manner by selecting an
item which may reduce the uncertainty about other unrated items. One approach
[51] for doing this is to define criteria by measuring the uncertainty of ratings over
all of the test items X(Test)with respect to a potential training input item xa:
b
GUncertainty(xa) = 1
X(Test)∑
x∈X(Test)
ET(a)(Uncertainty(Yx)),(24.10)
where 1
|X(Test)|is a normalizing factor, and ET(a)(Uncertainty(Yx)) is the expected
value of uncertainty with respect to adding an estimated rating yaof a candidate
item xato the training set T; i.e. T(a)=T∪(xa,ya).
A possible drawback of this non-local approach is that while with the local ap-
proach it is only necessary to estimate the uncertainty of a single output value ya,
for the non-local approach uncertainty needs to be estimated for the output values
of all the test points with respect to a potential training point (xa,ya); this may be
difficult to estimate accurately and could be computationally expensive.
24.4.1.2 Uncertainty Measurement
Uncertainty of an item’s rating (output value) is often measured by its variance, its
entropy [37], or by its confidence interval [50]. Variance is maximized when ratings
deviate the most from the mean rating, and entropy when all the ratings are equally
likely.
Uncertainty of an output value could be calculated by using a definition of vari-
ance as follows:
Uncertaint y(Ya) = VAR(Ya) = ∑
y∈Yy−Ya2P(Ya=y),(24.11)
where Yais the mean rating of all users for an item xaand P(Ya=y)is the proba-
bility of an items rating Yabeing equal to y, both being calculated based on either
nearest-neighbors for instance-based, or obtained from the model for model-based
approaches.
Uncertainty could also be measured by entropy as follows:
Uncertaint y(Ya) = ENT (Ya) = −∑
y∈Y
P(Ya=y)logP(Ya=y).(24.12)
In [58] a method is proposed for measuring the uncertainty of a rating based on the
probability of the most likely rating:
Uncertaint y(Ya) = −P(Ya=y∗),(24.13)
where y∗=argmaxyP(Ya=y)is the most likely rating.

24 Active Learning in Recommender Systems 833
In [50] the confidence interval is used as a measure of uncertainty for selecting
the training input point:
c=P(bl(Ya)<ya<bu(Ya)),(24.14)
where cis the confidence that the actual rating yawill lie in the interval between the
lower bound bl(Ya)and the upper bound bu(Ya). For example, it is possible for the
system to be certain that an item will be assigned a rating between 3 and 5 with a
probability c=90%. Many methods prefer items with a higher upper bound, indi-
cating that an item may be rated highly (good for exploitation), and if the confidence
interval is also wide then it may be good for exploration. In some cases where it is
desirable to increase the number of items predicted to be more highly rated, it may
be beneficial to use the expected change in the lower bound of the confidence inter-
val for selecting an item [50], the higher the expected change the more desirable.
24.4.2 Decision Boundary Uncertainty
(a) (b) (c)
candidate training input point
current decision boundary
new/refined decision boundary
Fig. 24.3: Decision boundary uncertainty.
In Decision Boundary-based methods, training points are selected so as to im-
prove decision boundaries. Often an existing decision boundary is assumed to be
somewhat accurate, so points are sampled close to the decision boundary to further
refine it (Figure 24.3 on page 833). In a way this may also be considered Output
Uncertainty-based, since the uncertainty of the points close to the decision boundary
may be high. This method operates with the assumption that the decision boundary
of the underlying learning method (e.g. Support Vector Machine) is easily accessi-
ble. A clear advantage of this method is that given a decision boundary, selecting
training examples by their proximity to it is computationally inexpensive.
As discussed in [56], training points may be selected for obtaining a more ac-
curate dividing hyperplane (Figure 24.3 on page 833 (b)), or if the direction of the

834 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
hyperplane is already certain, input points may be selected for reducing the size of
margin (Figure 24.3 on page 833 (c)). While it may seem obvious to sample training
points closest to the decision boundary [67, 16], there are also methods that select
the items furthest away [16] that have potential advantages in scenarios involving
several candidate classifiers, which are discussed in Section 24.6. This is because a
classifier should be quite certain about any items far from a decision boundary, but
if newly acquired training data reveals the classifier to be inaccurate, the classifier
may not fit the user’s preferences well, so it should be removed from the pool of
candidate classifiers.
24.4.3 Model Uncertainty
Model Uncertainty-based methods select training points for the purpose of reduc-
ing uncertainty within the model, more specifically to reduce uncertainty about
the model’s parameters. The assumption is that if we improve the accuracy of the
model’s parameters the accuracy of output values will improve as well. If we were to
predict a user’s preferences based on membership in different interest groups [29],
i.e. a group of people with a similar interest, then training points may be selected so
as to determine to which groups the user belongs (Section 24.4.3.1).
24.4.3.1 Probabilistic Models
Probabilistic models are best explained with an example. The aspect model [29], a
probabilistic latent semantic model in which users are considered to be a mixture of
multiple interests (called aspects) is a good choice for this. Each user u∈Uhas a
probabilistic membership in different interest groups z∈Z. Users in the same inter-
est group are assumed to have the same rating patterns (e.g. two users of the same
aspect will rate a given movie the same), so users and items x∈Xare indepen-
dent from each other given the latent class variable z. The probability of the user u
assigning an item xthe rating ycan be computed as follows:
P(y|x,u) = ∑
z∈Z
p(y|x,z)p(z|u).(24.15)
The first term p(y|x,z)is the likelihood of assigning an item xthe rating yby users
in class z(approximated by a Gaussian distribution in [29]). It does not depend on
the target user and represents the group-specific model. The global-model consists
of a collection of group-specific models. The second term p(z|u)is the likelihood
for the target user uto be in class z, referred to as a user personalization parameter
(approximated by a multinomial distribution in [29]). The user model θ
θ
θuconsists of
one or more user personalization parameters, i.e. θ
θ
θu={θuz=p(z|u)}z∈Z.
A traditional AL approach would be to measure the usefulness of the candidate
training input point xabased on how much it would allow for reduction of the un-

24 Active Learning in Recommender Systems 835
certainty about the user model’s parameters θ
θ
θu(i.e. the uncertainty about to which
interest group zthe user ubelongs):
b
GθUncert ainty (xa) = U ncertainty(θ
θ
θu),(24.16)
Uncertaint y(θ
θ
θu) = −*∑
z∈Z
θuz|xa,ylogθuz|xa,y+p(y|xa,θ
θ
θu)
,(24.17)
where θ
θ
θudenotes the currently estimated parameters of the user uand θuz|x,ya
parameter that is estimated using an additional training point (xa,y). Since the goal
of the above criterion is to reduce the uncertainty of which interest groups the target
user belongs to, it favors training points that assign a user to a single interest group.
This approach may not be effective for all models, such as with the aspect model, in
which a user’s preferences are better modeled by considering that a user belongs to
multiple interest groups [29, 31].
true decision boundary
estimated boundary
input2
input1
Fig. 24.4: A learning scenario when the estimated model is far from the true model.
Training points are indicated by solid contours.
Another potential drawback comes from the expected uncertainty being com-
puted over the distribution p(y|x,θ
θ
θu)by utilizing the currently estimated model θ
θ
θu.
The currently estimated model could be far from the true model, particularly when
the number of training points is small, but the number of parameters to be estimated
is large. Therefore, performing AL based only on a single estimated model can be
misleading [31]. Let us illustrate this by the following example shown in Figure
24.4 on page 835. The four existing training points are indicated by solid line con-
tours, test points by dashed ones. Based on these four training examples, the most
likely decision boundary is the horizontal line (dashed), even though the true deci-
sion boundary is a vertical line (solid). If we select training input points based only
on the estimated model, subsequent training points would likely be obtained from

836 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
areas along the estimated boundary, which are ineffective in adjusting the estimated
decision boundary (horizontal line) towards the correct decision boundary (vertical
line). This example illustrates that performing AL for the currently estimated model
without taking into account the model’s uncertainty can be very misleading, partic-
ularly when the estimated model is far from the true model. A better strategy could
be to consider model uncertainty by utilizing the model distribution for selecting
training input points [31]. This would allow for adjusting the decision boundary
more effectively since decision boundaries other than the estimated one (i.e. hori-
zontal line) would be considered for selecting the training input points. This idea is
applied to probabilistic models in [31] as follows. The usefulness of the candidate
training input point is measured based on how much it allows adjusting the model’s
parameters θ
θ
θutowards the optimal model parameters θ
θ
θu∗:
b
GθUncert ainty (xa) = *∑
z∈Z
θu∗
zlog θuz|xa,y
θu∗
z+p(y|xa,θ
θ
θu∗)
.(24.18)
The above equation corresponds to Kullback–Leibler divergence which is mini-
mized when the estimated parameters are equal to the optimal parameters. The true
model θ
θ
θu∗is not known but could be estimated as the expectation over the posterior
distribution of the user’s model i.e. p(θ
θ
θu|u).
24.5 Error-based Active Learning
Error-based Active Learning methods aim to reduce the predictive error, which is
often the final goal. Instance-based approaches try to find and utilize the relation
between the training input points and the predictive error. Model-based approaches
tend to aim at reducing the model error (i.e. the error of model parameters), which
is hoped would result in the improvement of predictive error.
24.5.1 Instance-based Methods
Instance-based methods aim at reducing error based on the properties of the input
points, such as are listed in Section 24.2.
24.5.1.1 Output Estimates Change (Y-Change)
This approach [53] operates on the principle that if rating estimates do not change
then they will not improve. Thus, if the estimates of output values do change, then
their accuracy may either increase or decrease. However, it is expected that at least
something will be learned from a new training point, so it follows then that in many

24 Active Learning in Recommender Systems 837
cases estimates do in fact become more accurate. Assuming that most changes in
estimates are for the better, an item that causes many estimates to change will result
in the improvement of many estimates, and is considered useful.
(a)
(b)
Fig. 24.5: Output estimate-based AL (Section 24.5.1.1). The x-axis corresponds to
an item’s index, and the y-axis to the changes in rating estimates with regard to a
candidate training point. Training points that cause many changes in rating estimates
are considered to be more informative (a).
As an example (Figure 24.5 on page 837), if a user rates an item that is represen-
tative of a large genre, such as the Sci-Fi movie Star Wars, then its rating (regard-
less of its value) will likely cause a change in rating estimates for many other related
items (e.g. items within that genre), in other words, rating such a representative item
is very informative about the user’s preferences. On the other hand, the user rating
an item without many other similar items, such as the movie The Goldfish Hunter,
would change few rating estimates, and supply little information.
To find the expected changes in rating estimates caused by a candidate item’s
rating, all possible item ratings are considered (since the true rating of a candidate
item is not yet known). The difference is calculated between rating estimates for
each item for each of its possible ratings, before and after it was added to the training
set (refer to the pseudocode in 1).
More formally the above criterion could be expressed as:

838 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
Algorithm 1 Output estimates-based Active Learning (Section 24.5.1.1).
#b
Gestimates predictive error that rating an item xawould allow to achieve
function b
G(xa)
# learn a preference approximation function b
fbased on the current training set T
b
fT=learn(T)
# for each possible rating of an item xae.g. {1,2,...,5}
for ya∈Y
# add a hypothetical training point (xa,ya)
T(a)=T∪(xa,ya)
# learn a new preference approximation function b
fbased on the new training set T(a)
b
fT(a)=learn(T(a))
# for each unrated item
for x∈X(Test)
# record the differences between ratings estimates
# before and after a hypothetical training point (xa,ya)was added to the training set T
b
G=b
G+−b
fT(x)−b
fT(a)(x)2
return b
G
b
GYchange (xa) = −∑
x∈X(Test)
Ey∈YL(b
fT(x),b
fT∪(xa,y)(x)),(24.19)
where b
fT(x)is the estimated rating for an item xgiven the current training set T,
and b
fT∪(xa,y)(x)is the rating’s estimate after a hypothetical rating yof an item xais
added to the training set T, and Lis the loss function that measures the differences
between the rating estimates b
fT(x)and b
fT∪(xa,y)(x). By assuming that ratings of a
candidate item are equally likely and using a mean squared loss function, the above
criterion could be written as:
b
GYchange (xa) = −∑
x∈X(Test)
1
|Y|∑
y∈Yb
fT(x)−b
fT∪(xa,y)(x)2(24.20)
where 1
|Y|is a normalizing constant since we assume all possible ratings y∈Yof an
item xa.
The advantage of this criterion is that it relies only on the estimates of ratings,
available from any learning method. It has a further advantage of utilizing all unrated
items, something that differentiates it from other methods in which only a small
subset of all items (ones that have been rated by the user) are considered. It also
works in tandem with any of a variety of learning methods, enabling it to potentially
adapt to different tasks.

24 Active Learning in Recommender Systems 839
24.5.1.2 Cross Validation-based
In this approach a training input point is selected based on how well it may allow for
approximation of already known ratings, i.e. items in the training set [16]. That is,
a candidate training point xawith each possible rating y∈Yis added to the training
set T, then an approximation of the user’s preferences b
fis obtained and its accuracy
is evaluated (i.e. cross-validated) on the training items X(Train). It is assumed that
when the candidate training item is paired with its correct rating, the cross-validated
accuracy will improve the most. The usefulness of the candidate training point is
measured by the improvement in the cross-validated accuracy as following:
b
GCVT(xa) = −max
y∈Y∑
x∈X(Train)
L(b
fT∪(xa,y)(x),f(x)),(24.21)
where Lis a loss function such as MAE or MSE (Section 24.3.1), and f(x)is the
actual rating of the item x, and b
fT∪(xa,y)(x)is the approximated rating (where a
function b
fis learned from the training set T∪(xa,y)) .
A potential drawback is that training points selected by this AL method could be
overfitted to the training set.
24.5.2 Model-based
In model-based approaches training input points are obtained as to reduce the
model’s error, i.e. the error of the model’s parameters. A potential drawback of this
approach is that reducing the model’s error may not necessarily reduce the predic-
tion error which is the objective of AL.
24.5.2.1 Parameter Change-based
Parameter Change-based AL [60] favors items that are likely to influence the model
the most. Assuming that changes in the model’s parameters are for the better, i.e.
approach the optimal parameters, it is then beneficial to select an item that has the
greatest impact on the model’s parameters:
b
Gθchange(xa) = −∑
θ
Ey∈YL(θT,θT∪(xa,y)),(24.22)
where θTare the model’s parameters estimated from the current training set T, and
θT∪(xa,y)are the model’s parameter estimates after a hypothetical rating yof an
item xais added to the training set T, and Lis the loss function that measures the
differences between the parameters.

840 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
24.5.2.2 Variance-based
solution space
V
C
G
variance
generalization error
model error
g
f
B
bias
^
f1
^
f3
^
f2
^
f
Fig. 24.6: Decomposition of generalization error Ginto model error C, bias B, and
variance V, where gdenotes optimal function, b
fis a learned function b
fi’s are the
learned functions from a slightly different training set.
In this approach the error is decomposed into three components: model error
C(the difference between the optimal function approximation g, given the current
model, and the true function f), bias B(the difference between the current approx-
imation b
fand an optimal one g), and variance V(how much the function approxi-
mation b
fvaries ). In other words, we have:
G=C+B+V.(24.23)
One solution [14] is to minimize the variance componentVof the error by assuming
that the bias component becomes negligible (if this assumption is not satisfied then
this method may not be effective). There are a number of methods proposed that
aim to select training inputs for reducing a certain measure of the variance of the
model’s parameters. The A-optimal design [12] seeks to select training input points
so as to minimize the average variance of the parameter estimates, the D-optimal
design [33] seeks to maximize the differential Shannon information content of the
parameter estimates, and the Transductive Experimental design [68] seeks to find
representative training points that may allow retaining most of the information of
the test points. The AL method in [62], in addition to the variance component, also
takes into account the existense of the model error component.
24.5.2.3 Image Restoration-based
It is also possible to treat the problem of predicting the user’s preferences as one
of image restoration [44], that is, based on our limited knowledge of a user’s pref-
erences (a partial picture), we try to restore the complete picture of the user’s likes
and dislikes. The AL task is then to select the training points that would best allow

24 Active Learning in Recommender Systems 841
us to restore the “image” of the user’s preferences. It is interesting to note that this
approach satisfies the desired properties of the AL methods outlined in Section 24.2.
For example, if a point already exists in a region, then without sampling neighbor-
ing points the image in that region could likely be restored. This approach also may
favor sampling close to the edges of image components (decision boundaries).
24.6 Ensemble-based Active Learning
Sometimes instead of using a single model to predict a user’s preferences, an ensem-
ble of models may be beneficial (Chapter 22). In other cases only a single model is
used, but it is selected from a number of candidate models. The main advantage of
this is the premise that different models are better suited to different users or dif-
ferent problems. The preferences of one user, for example, could be better modeled
by a stereotype model, while the preferences of another user may be better mod-
eled by a nearest-neighbor model. The training input points for these AL methods
must be selected with regards to multiple models (Section 24.6.1) or multiple model
candidates (Section 24.6.2).
24.6.1 Models-based
In Models-based approaches, the models form a “committee” of models that act,
in a sense, cooperatively to select training input points [61]. Methods tend to differ
with respect to: (1) how to construct a committee of models, and (2) how to se-
lect training points based on committee members [57]. As [57] explains thoroughly
(please refer to it for more details), the Query by Committee approach (QBC) in-
volves maintaining a committee of models which are all trained on the same training
data. In essence, they represent competing hypotheses for what the data might look
like (as represented by the model). The members of this committee then vote on how
to label potential input points (the “query” in “QBC”). The input points for which
they disagree the most are considered to be the most informative. The fundamental
premise of QBC is minimizing the version space, or the subset of all hypotheses that
are consistent with all the collected training data; we want to then constrain the size
of this space as much as possible, while at the same time minimizing the number of
training input points. Put a different way, QBC “queries” in controversial regions to
refine the version space.
There are many ways to construct the committee of models; [57] provides nu-
merous examples. It can, for example, be constructed through simple sampling [61].
With generative model classes, this can be achieved by randomly sampling an ar-
bitrary number of models from some posterior distribution, e.g. using the Dirich-
let distribution over model parameters for naive Bayes [41], or sampling Hidden
Markov Models (HMMs) using the Normal distribution [15]. The ensemble can

842 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
be constructed for other model classes (such as discriminative or non-probabilistic
models) as well, e.g. query-by-boosting and query-by-bagging [1], which employ
the boosting [22] and bagging [8] ensemble learning methods to construct the com-
mittees; there has also been research [13] on using a selective sampling algorithm
for neural networks that utilizes the combination of the “most specific” and “most
general” models (selecting the models that lie at two extremes of the current version
space given the current training set).
The “committee is still out” on the appropriate number of models to use, but even
small sizes have demonstrated good results [61, 41, 58].
Measuring the disagreement between models is fundamental to the committee
approach; there are two main means for calculating disagreement: vote uncertainty
[15] and average Kullback-Leibler (KL) divergence [41]. Vote uncertainty selects
the point with the largest disagreement between models of the committee. KL diver-
gence is an information-theoretic measure of the difference between two probability
distributions. KL divergence selects the input point with the largest average differ-
ence between the distributions of the committee consensus and the most differing
model.
24.6.2 Candidates-based
Different models are better suited to different users or to different problems (7). So
both the choice of the training set (AL) and the choice of the model, called Model
Selection (MS), affect the predictive accuracy of the learned function. There is in
fact a strong dependency between AL and MS, meaning that useful points for one
model may not be as useful for another (Figure 24.8 on page 844). This section
discusses how to perform AL with regards to multiple model candidates and the
issues that may arise when doing so.
The concept of model has several different meanings. We may refer to a model as
a set of functions with some common characteristic, such as a function’s complex-
ity, or the type of a function or learning method (e.g. SVM, Naive Bayes, nearest-
neighbor, or linear regression). The characteristics of the functions that may differ
are often referred to as parameters. Thus, given a model and training data, the task
of MS is to find parameters that may allow for accurate approximation of the target
function. All of the model’s characteristics affect the predictive accuracy, but for
simplicity we concentrate only on the complexity of the model.
As illustrated by Figure 24.7 on page 843, if the model is too simple in com-
parison with the target function, then the learned function may not be capable of
approximating the target function, making it under-fit (Figure 24.7a on page 843).
On the other hand, if the model is too complex it may start trying to approximate
irrelevant information (e.g. noise that may be contained in the output values) which
will cause the learned function to over-fit the target function (Figure 24.7b on page
843). A possible solution to this is to have a number of candidate models. The goal
of model selection (MS) is thus to determine the weights of the models in the en-

24 Active Learning in Recommender Systems 843
(a) under-fit (b) over-fit (c) appropriate fit
Fig. 24.7: Dependence between model complexity and accuracy.
semble, or in the case of a single model being used, to select an appropriate one
(Figure 24.7c on page 843): min
MG(M).(24.24)
The task of AL is likewise to minimize the predictive error, but with respect to the
choice of the training input points:
min
X(Train)G(X(T rain)).(24.25)
It would be beneficial to combine AL and MS since they share a common goal of
minimizing the predictive error:
min
X(Train),M
G(X(Train),M).(24.26)
Ideally we would like to choose the model of appropriate complexity by a MS
method and to choose the most useful training data by an AL method. However
simply combining AL with MS in a batch manner, i.e. selecting all of the training
points at once, may not be possible due to the following paradox:
•To select training input points by a standard AL method, a model must be fixed.
In other words, MS has already been performed (see Figure 24.8 on page 844).
•To select the model by a standard MS method, the training input points must
be fixed and corresponding training output values must be gathered. In other
words, AL has already been performed (see Figure 24.9 on page 844).
As a result Batch AL selects training points for a randomly chosen model, but
after the training points are obtained the model is selected once again, giving rise
to the possibility that the training points will not be as useful if the initial and final
models differ. This means that the training points could be over-fitted to a possibly
inferior model, or likewise under-fitted.
With Sequential AL, the training points and models are selected incrementally
in a process of selecting a model, then obtaining a training point for this model,
and so on. Although this approach is intuitive, it may perform poorly due to model
drift, where a chosen model varies throughout the learning process. As the number
of training points increases, more complex models tend to fit data better and are
therefore selected over simpler models. Since the selection of training input points

844 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
Fig. 24.8: Training input points that are good for learning one model, are not neces-
sary good for the other.
Unable to determine which model is more appropriate (Model Selec-
tion), until training points have been obtained (Active Learning).
Fig. 24.9: Dependence of Model Selection on Active Learning.
select model
MS
select training point
for model
AL
select final model
MS
more
points?
yes
no
Fig. 24.10: Sequential Active Learning.
select model
MS
select all training points
for model
AL
select final model
MS
Fig. 24.11: Batch Active Learning.

24 Active Learning in Recommender Systems 845
depends on the model, the training points chosen for a simpler model in the early
stages could be less useful for the more complex model selected at the end of the
learning process. Due to model drift portions of training points are gathered for
different models, resulting in the training data being not well suited for any of the
models. However, because the selection of the final model is unclear at the onset,
one possibility is to select training input points with respect to multiple models [63],
by optimizing the training data for all the models:
min
X(Train)∑
Mb
G(X(Train),M)w(M),(24.27)
where w(M)refers to the weight of the model in the ensemble, or among the candi-
dates. This allows each model to contribute to the optimization of the training data
and thus the risk of overfitting the training set to possibly inferior models can be
hedged.
24.7 Conversation-based Active Learning
Preference elicitation [45], just as active learning, aims at constructing an accurate
model of user preferences. However, unlike AL that elicits ratings to inductively
model the user preferences, preference elicitation aims to lean about user prefer-
ences on a more abstract level e.g. by directly acquiring or deducting user’s prefer-
ences (e.g. by asking users which genre of movies they like). Preference elicitation
is often performed with the help of conversation-based AL that is goal oriented with
the task of starting general and, through a series of interaction cycles, narrowing
down the user’s interests until the desired item is obtained [42, 49, 9], such as se-
lecting a hotel to stay at during a trip. In essence, the goal is to supply the user with
the information that best enables them to reduce the set of possible items, finding the
item with the most utility. The system therefore aims at making accurate predictions
about items with the highest utility for a potentially small group of items, such as
searching for a restaurant within a restricted locale. A common approach is to itera-
tively present sets of alternative recommendations to the user, and by eliciting feed-
back, guide the user towards an end goal in which the scope of interest is reduced to
a single item. This cycle-based approach can be beneficial since users rarely know
all their preferences at the start (becoming self-aware), but tend to form and refine
them during the decision making process (exploration). Thus Conversation-based
AL should also allow users to refine their preferences in a style suitable to the given
task. Such systems, unlike general RSs, also include AL by design, since a user’s
preferences are learned through active interaction. They are often evaluated by the
predictive accuracy, and also by the length of interaction before arriving at the de-
sired goal.

846 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
24.7.1 Case-based Critique
One means for performing a conversation with a user is the Case-based Critique
approach, which finds cases similar to the user’s query or profile and then elicits
a critique for refining the user’s interests [49]. As mentioned above (Section 24.7),
the user is not required to clearly define their preferences when the conversation ini-
tiates; this may be particularly beneficial for mobile device-oriented systems. Each
step of iteration displays the system’s recommendations in a ranked list and allows
for user critique, which will force the system to re-evaluate its recommendations
and generate a new ranked list. Eliciting a user critique when a feature of a recom-
mended item is unsatisfactory may be more effective in obtaining the end goal than
mere similarity-based query revision combined with recommendation by propos-
ing. As an example of a user critique, he/she may comment “I want a less expensive
hotel room” or “I like restaurants serving wine.”
24.7.2 Diversity-based
While suggesting items to the user that are similar to the user query is important
(24.7.1), it may also be worthwhile to consider diversity among the set of proposed
items [42]. This is because if the suggested items are too similar to each other, they
may not be representative of the current search space. In essence, the recommended
items should be as representative and diverse as possible, which should be possi-
ble without appreciably affecting their similarity to the user query. It is particularly
important to provide diverse choices while the user’s preferences are in their embry-
onic stages. Once the user knows what it is they want, providing items that match as
closely as possible may be pertinent, and the AL technique used should attempt to
make this distinction, i.e. if the recommendation space is properly focused, reduce
diversity, and if incorrect, increase it.
24.7.3 Query Editing-based
Another possibility is to allow a user to repeatedly edit and resubmit a search query
until their desired item is found [9]. Since it is an iterative process, the objective is
to minimize the number of queries needed before the user finds the item of highest
utility. A query’s usefulness is estimated based on the likelihood of the user submit-
ting a particular query, along with its satisfiability, accomplished by observing user
actions and inferring any constraints on user preferences related to item utility and
updating the user’s model. As an example, a user may query for hotels that have
air-conditioning and a golf course. The RS can determine this to be satisfiable, and
further infer that though the user is likely to add a restraint for the hotel being lo-
cated in the city-center, no hotels match such criteria, so the system preemptively

24 Active Learning in Recommender Systems 847
notifies the user that such a condition is unsatisfiable to prevent wasted user effort.
The RS may also infer that for a small increase in price there are hotels with a pool
and spa and a restaurant. Knowing the user’s preferences for having a pool (and not
for other options), the system would only offer adding the pool option, since it may
increase the user’s satisfaction, and not the others since they may overwhelm the
user and decrease overall satisfaction.
24.8 Evaluation Settings
Proper evaluation of active learning is important for measuring how well the system
meets given objectives, and investigating if there are any undesirable side effects.
In experimental studies, evaluation setup should reflect as closely as possible the
actual settings for which the system was designed. In Section 24.3.1 we briefly
described machine learning-based settings under which active learning algorithms
are typically evaluated. If one aims for a more realistic evaluation, other domain
specific aspects must be considered (some of which are described below).
24.8.1 Scope
rating values (proportional to size)
ratings known by the system (K)
ratings known by the user
but not the system (X)
System can elicit a rating
ratings used for evaluation (T)
items items
users
users
(a) User-centered Active Learning (b) System-centered Active Learning
Legend
Fig. 24.12: Comparison of the scope of the ratings data configurations used for eval-
uating user-centered and system-centered active learning strategies (Section 24.8.1).
Traditionally, the evaluation of active learning strategies has been user-centered;
that is, the usefulness of the elicited rating was judged based on the improvement
in the user’s prediction error. This is illustrated in Figure 24.12(a). In this scenario
the system is supposed to have a large number of ratings from several users, and
focusing on a new user (the first one in Figure 24.12(a)) it first elicits ratings from

848 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
this new user that are in X, and then system predictions for this user are evaluated on
the test set T . Hence these traditional evaluations focussed on the new-user prob-
lem and measured how the ratings elicited from a new user may help the system to
generate good recommendations for this particular user. [20] noted that eliciting a
rating from a user may improve not only the rating predictions for that user, but also
the predictions for the other users, as is graphically illustrated in Figure 24.12(b).
To illustrate this point, let us consider an extreme example in which a new item
is added to the system. The traditional user-centered AL strategy, when trying to
identify the items that a target user should rate, may ignore obtaining user’s rating
for that new item. In fact, this item has not been rated by any other user and there-
fore its ratings cannot contribute to improving the rating predictions for the target
user. However, the rating of the target user for the new item would allow to boot-
strap the predictions 8for the rest of the users in the system, and hence from the
system’s perspective the elicited rating is indeed very informative. Conversely, it
was shown some user-centric strategies, while being beneficial for the target user,
may increasing the system error. For instance, requesting to rate the items with the
highest predicted ratings (an AL approach that is often adopted in real RSs), may
generate a system-wide bias, and inadvertently increase the system error (especially
at the early stages) by adding to the training set disproportionately more high ratings
than low ones, and as a result biasing the rating prediction towards overestimating
ratings. Elicited rating has effects across the system, so a typical user-centric eval-
uation which ignores any changes of rating prediction of other users also ignores
these cumulative effects, which could be more influential on the performance of the
system as a whole.
24.8.2 Natural Rating Acquisition
In RSs there are two primary ways in which ratings are acquired: (1) users are
prompted to rate an item by an active learning method; (2) users provide ratings
without being prompted (natural rating acquisition), e.g. while browsing items. Pre-
vious studies considered the situation where the active learning rating elicitation
strategy was the only tool used to collect new ratings from the users. Recently, [20]
has proposed a more realistic evaluation setting, where in addition to the ratings
being acquired by the elicitation strategies, the ratings are also entered by the users
(without being prompted), similarly to what happens in actual settings. Mixing in
naturally acquired ratings significantly impacts the performance of some of the ac-
tive learning methods (Section 24.8.5). For example, without mixing in naturally
acquired ratings, highest-predicted AL is shown to acquire many new ratings. Yet,
when the naturally acquired ratings are mixed in, highest-predicted AL acquires
very few ratings since many of the ratings are already collected by the natural pro-
cess (i.e. the user would rate these items on his own initiative).
8Recently it has also been proposed to utilize transfer learning for leveraging pre-existing labeled
data from related tasks to improve the performance of an active learning algorithm [69, 34].

24 Active Learning in Recommender Systems 849
24.8.3 Temporal Evolution
Ratings data changes with time: more ratings are added, new users and items appear,
underlying recommendation and active learning algorithms change, as do user inter-
faces. While it is convenient to evaluate AL method on a snapshot of the database;
it is also advisable to incorporate temporal aspects of RSs in order to obtain a more
complete view of the algorithm’s performance. proposed considering temporal as-
pects with a simulation loop that models the day-by-day process of rating elicitation
and rating database growth (starting from an empty database); where users repeat-
edly come back to the system for receiving recommendations, while the system has
possibly elicited ratings from other users. To achieve a realistic setting, only the
items that users actually experienced during the following week (according to the
timestamps) are added to the database for each time period. [20] showed that differ-
ent strategies improve different aspects of the recommendation quality at different
stages of the rating database growth. Moreover, performance of AL varies signifi-
cantly from week to week, caused by the fact that for every week system is trained
on the data from previous weeks, and is evaluated on the next week’s ratings. Hence,
the quality of the training data and predictive difficulty of the test set can therefore
change from week to week, and hence influence the performance of the AL strate-
gies. [70] proposed AL method that explicitly takes temporal changes into account,
focusing on changes in users preferences over time.
Time-dependent evolution of predictive aspects of recommender systems has also
received some attention. In [10] the author analyzes the temporal properties of a
standard user-based collaborative filtering [26] and Influence Limiter [48], a collab-
orative filtering algorithm developed for counteracting profile injection attacks by
considering the time at which a user has rated an item. These works evaluate the ac-
curacy of prediction algorithms while the users are rating items and the database is
growing. This is radically different from the typical evaluations that we mentioned
earlier, where the rating dataset is decomposed into the training and testing sets with-
out considering the timestamp of the ratings. In [10] it is argued that considering the
time at which the ratings were added to the system gives a better picture of the real
user experience during the interactions with the system in terms of recommendation
accuracy. They discovered the presence of two time segments: the start-up period,
until day 70 with MAE dropping gradually, and the remaining period, where MAE
was dropping much slower.
24.8.4 Ratability
A user may not always be able to provide a rating for an item; e.g. you cannot rate a
movie you have not seen [25, 38]. On the other hand, the system typically contains
ratings for only a portion of items that users have experienced. This is a common
problem of any offline evaluation of a recommender system, where the performance
of the recommendation algorithm is estimated on a test set that is never coincident

850 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
with the recommendations set. The recommendation set is composed of the items
with the largest predicted ratings. But if such an item is not present in the test set,
an offline evaluation will be never able to check if that prediction is correct [19].
Only a few evaluations simulated limited knowledge about the user’s ratings ([25],
[20]); from the user and/or system perspective. The system is assumed to unaware
about what items the simulated user has experienced, and may ask ratings for items
that the user will not be able to provide. This better simulates a realistic scenario
where not all rating requests can be satisfied by a user. It is important to note that
the simulated application of an active learning strategy is able to add many fewer
ratings than what could be elicited in a real setting. In fact, the number of ratings
that are supposed to be known by the users in the simulated process is limited by the
number of ratings that are present in the dataset. In [19] it has been estimated that the
number of items that are really known by the user is more than 4 times larger than
what is typically observed in the simulations. Hence, a lot of our elicitation request
would be unfulfilled, even though the user in actuality would have been able to rate
the item. To adjust for the discrepancy between the knowledge of the actual and
simulated users, it is recommended to increase number of active learning requests
by a factor of 4 [19].
24.8.5 Summary
Table 24.3: Summary of Performance Evaluation (performance: !– good, #– bad.)
39:28 M. Elahi et al.
0 5 10 15 20 25 30 35 40 45 50
0
0.5
1
1.5
2
2.5
3
3.5 x 105
# of weeks
# of ratings in Known set
natural
variance
random
popularity
lowest−pred
lo−hi−pred
highest−pred
binary−pred
voting
log(pop)*entropy
Fig. 14. Size evolution of the Known set under the application of rating elicitation strategies (Movielens).
Table III. Strategies Performance Summary (performance: - good, .- bad)
Strategies Metrics
MAE NDCG Elicited # Inform. Precision
Early Stage
Late Stage
Randomized
w/ Natural
Early Stage
Late Stage
Randomized
Early Stage
Late Stage
w/ Natural
Early Stage
Late Stage
Early Stage
Late Stage
Randomized
variance
popularity
lowest-pred
lo-hi-pred
highest-pred
binary-pred
voting
log(pop)*ent
random NA NA NA
natural NA NA
highest-lowest-predicted. Moreover, we have studied the behavior of other novel strategies: partially
randomized, which adds random ratings in the elicitation lists computed by the aforementioned
strategies; voting, which requests to rate the items that are selected by the largest number of voting
strategies. We have evaluated these strategies with regards to their system-wide effectiveness by
implementing a simulation loop that models the day-by-day process of rating elicitation and rating
database growth. We have taken into account the limited knowledge of the users, which means that
the users may not be able to rate all the items that the system proposes them to rate. During the
simulation we have measured several metrics at different phases of the rating database growth. The
metrics include: MAE to measure the improvements in prediction accuracy, precision to measure
the relevance of recommendations, normalized discounted cumulative gain (NDCG) to measure the
ACM Transactions on Intelligent Systems and Technology, Vol. 9, No. 4, Article 39, Publication date: March 2010.
In [20] performance of many of the common active learning methods has been
evaluated considering many of the aspects mentioned above, as to more realistically
simulate actual RS settings. The evaluation (summarized in Table 24.3 on page 850)

24 Active Learning in Recommender Systems 851
has shown that the system-wide effectiveness of a rating elicitation strategy (Sec-
tion 24.8.1) depends on the stage of the rating elicitation process (Section 24.8.3),
and on the evaluation metrics (Section 24.2, Section 24.1.1). Surprisingly, some
common user-centric strategies (Section 24.8.1) may actually degrade the overall
performance of a system. Finally, the performance of many common active learning
strategies changes significantly when evaluated concurrently with e.g. the natural
acquisition of ratings (Section 24.8.2).
24.9 Computational Considerations
It is also important to consider the computational costs of AL algorithms. [51] have
suggested a number of ways of reducing the computational requirements, summa-
rized (with additions) below.
•Many AL select an item to be rated based on its expected effect on the learned
function. This may require retraining with respect to each candidate training
item, and so efficient incremental training is crucial. Typically this step-by-step
manner has lower cost than starting over with a large set.
•New rating estimates may need to be obtained with respect to each candidate
item. Likewise, this could be done in an incremental manner, since only the
estimates that change would need to be obtained again.
•It is possible to incrementally update the estimated error only for items likely to
be effected by the inclusion of a training point, which in practice is only nearby
items or items without similar features. A common approach is to use inverted
indices to group items with similar features for quick lookup.
•A candidate training item’s expected usefulness can likely be estimated using a
subset of all items.
•Poor candidates for training points can be partially pruned through a pre-
filtering step that removes poor candidate items based on some criteria, such
as filtering books written in a language the user cannot read. A suboptimal AL
method may be a good choice for this task.
24.10 Discussion
Though very brief, hopefully the collection of Active Learning methods presented
in this chapter has demonstrated that AL is indeed not only beneficial but also desir-
able for inclusion in many systems, namely Recommender Systems. It can be seen
that due to individual characteristics, the AL method selected, in many cases, relies
heavily on the specific objectives (Section 24.1.1) that must be satisfied, either due
to business constraints, preferred system behavior, user experience, or a combina-
tion of these (and possibly others). In addition to AL objectives, it is also prudent

852 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
to evaluate the computational costs (Section 24.9) of any methods under consid-
eration for use, and their trade-offs. Despite the success that many of the meth-
ods discussed have received, there is also something to be said for abstracting the
problem, or finding solutions to other problems that though seemingly unrelated,
may have strikingly similar solutions (e.g. Image Restoration (Section 24.5.2.3)).
We have also touched upon conversation-based systems (Section 24.7) which differ
from traditional RSs, but include the notion of AL by design. Depending on the task
at hand, such as specific goal oriented assistants, this may also be a nice fit for a
Recommender System.
Some issues related to AL have already been well studied in Statistics; this is
not the case in Computer Science, where research is still wanting. Recommender
Systems are changing at a rapid pace and becoming more and more complex. An
example of this is the system that won the NetFlix Recommendation Challenge,
which combined multiple predictive methods in an ensemble manner (Chapter 3).
Given the high rate of change in predictive methods of RSs, and their complex
interaction with AL, there is an ever increasing need for new approaches.
Improving accuracy has traditionally been the main focus of research. Accuracy
alone, however, may not be enough to entice the user with RSs (Chapter 8). This is
because the system implementing AL may also need to recommend items of high
novelty/serendipity, improve coverage, or maximize profitability, to name a few [55,
27, 43, 32]. Another aspect that is frequently overlooked by AL researchers is the
manner in which a user can interact with AL to reap improvements in performance.
Simply presenting items to the user for rating lacks ingenuity to say the least; surely
there is a better way? One example of this is a work [3] which demonstrated that by
using the right interface even such menial tasks as labeling images could be made
fun and exciting. With the right interface alone the utility of an AL system may
increase dramatically.
Many issues remain that must be tackled to ensure the longevity of AL in RSs;
with a little innovation and elbow grease we hope to see it transform from a “both-
ersome process” to an enjoyable one of self-discovery and exploration, satisfying
both the system objectives and the user at the same time.
References
1. Abe, N., Mamitsuka, H.: Query learning strategies using boosting and bagging. In: Proceed-
ings of the Fifteenth International Conference on Machine Learning, vol. 388. Morgan Kauf-
mann Publishers Inc. (1998)
2. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: A survey
of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data
Engineering 17(6), 734–749 (2005)
3. Ahn, L.V.: Games with a purpose. Computer 39(6), 92–94 (2006). DOI 10.1109/MC.2006.196
4. Bailey, R.A.: Design of Comparative Experiments. Cambridge University Press (2008)
5. Balcan, M.F., Beygelzimer, A., Langford, J.: Agnostic active learning. In: ICML ’06: Pro-
ceedings of the 23rd international conference on Machine learning, pp. 65–72. ACM, New
York, NY, USA (2006). DOI http://doi.acm.org/10.1145/1143844.1143853

24 Active Learning in Recommender Systems 853
6. Boutilier, C., Zemel, R., Marlin, B.: Active collaborative filtering. In: Proceedings of the
Nineteenth Annual Conference on Uncertainty in Artificial Intelligence, pp. 98–106 (2003).
URL citeseer.ist.psu.edu/boutilier03active.html
7. Box, G., Hunter, S.J., Hunter, W.G.: Statistics for Experimenters: Design, Innovation, and
Discovery. Wiley-Interscience (2005)
8. Breiman, L., Breiman, L.: Bagging predictors. In: Machine Learning, pp. 123–140 (1996)
9. Bridge, D., Ricci, F.: Supporting product selection with query editing recommendations. In:
RecSys ’07: Proceedings of the 2007 ACM conference on Recommender systems, pp. 65–72.
ACM, New York, NY, USA (2007). DOI http://doi.acm.org/10.1145/1297231.1297243
10. Burke, R.: Evaluating the dynamic properties of recommendation algorithms. In: Proceed-
ings of the fourth ACM conference on Recommender systems, RecSys ’10, pp. 225–228.
ACM, New York, NY, USA (2010). DOI http://doi.acm.org/10.1145/1864708.1864753. URL
http://doi.acm.org/10.1145/1864708.1864753
11. Carenini, G., Smith, J., Poole, D.: Towards more conversational and collaborative recom-
mender systems. In: IUI ’03: Proceedings of the 8th international conference on Intelligent
user interfaces, pp. 12–18. ACM, New York, NY, USA (2003). DOI http://doi.acm.org/10.
1145/604045.604052
12. Chan, N.: A-optimality for regression designs. Tech. rep., Stanford University, Department of
Statistics (1981)
13. Cohn, D.A.: Neural network exploration using optimal experiment design 6, 679–686 (1994).
URL citeseer.ist.psu.edu/article/cohn94neural.html
14. Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. Journal of
Artificial Intelligence Research 4, 129–145 (1996)
15. Dagan, I., Engelson, S.: Committee-based sampling for training probabilistic classifiers. In:
Proceedings of the International Conference on Machine Learning (ICML), pp. 150–157. Cite-
seer (1995)
16. Danziger, S., Zeng, J., Wang, Y., Brachmann, R., Lathrop, R.: Choosing where to look next in
a mutation sequence space: Active learning of informative p53 cancer rescue mutants. Bioin-
formatics 23(13), 104–114 (2007)
17. Dasgupta, S., Lee, W., Long, P.: A theoretical analysis of query selection for collabora-
tive filtering. Machine Learning 51, 283–298 (2003). URL citeseer.ist.psu.edu/
dasgupta02theoretical.html
18. Diaz-Aviles, E., Drumond, L., Schmidt-Thieme, L., Nejdl, W.: Real-time top-n recommen-
dation in social streams. In: Proceedings of the Sixth ACM Conference on Recommender
Systems, RecSys ’12, pp. 59–66. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.
2365968. URL http://doi.acm.org/10.1145/2365952.2365968
19. Elahi, M.: Adaptive active learning in recommender systems. In: User Modeling, Adaption
and Personalization - 19th International Conference, UMAP 2011, Girona, Spain, July 11-15,
2011. Proceedings, pp. 414–417 (2011)
20. Elahi, M., Ricci, F., Rubens, N.: Active learning strategies for rating elicitation in collabo-
rative filtering: a system-wide perspective. ACM Transactions on Intelligent Systems and
Technology 5(11) (2013)
21. Ertekin, S., Huang, J., Bottou, L., Giles, L.: Learning on the border: active learning in imbal-
anced data classification. In: Proceedings of the sixteenth ACM conference on Conference on
information and knowledge management, pp. 127–136. ACM (2007)
22. Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an appli-
cation to boosting. Journal of computer and system sciences 55(1), 119–139 (1997)
23. Fujii, A., Tokunaga, T., Inui, K., Tanaka, H.: Selective sampling for example-based word sense
disambiguation. Computational Linguistics 24, 24–4 (1998)
24. Greiner, R., Grove, A., Roth, D.: Learning cost-sensitive active classifiers. Artificial Intelli-
gence 139, 137–174 (2002)
25. Harpale, A.S., Yang, Y.: Personalized active learning for collaborative filtering. In: SIGIR
’08: Proceedings of the 31st annual international ACM SIGIR conference on Research and

854 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
development in information retrieval, pp. 91–98. ACM, New York, NY, USA (2008). DOI
http://doi.acm.org/10.1145/1390334.1390352
26. Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for per-
forming collaborative filtering. In: Proceedings of the 22nd annual international ACM SIGIR
conference on Research and development in information retrieval, SIGIR ’99, pp. 230–237.
ACM, New York, NY, USA (1999). DOI http://doi.acm.org/10.1145/312624.312682. URL
http://doi.acm.org/10.1145/312624.312682
27. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering
recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). DOI http://doi.acm.org/10.
1145/963770.963772
28. Hinkelmann, K., Kempthorne, O.: Design and Analysis of Experiments, Advanced Experi-
mental Design. Wiley Series in Probability and Statistics (2005)
29. Hofmann, T.: Collaborative filtering via gaussian probabilistic latent semantic analysis. In:
SIGIR ’03: Proceedings of the 26th annual international ACM SIGIR conference on Research
and development in informaion retrieval, pp. 259–266. ACM, New York, NY, USA (2003).
DOI http://doi.acm.org/10.1145/860435.860483
30. Huang, Z.: Selectively acquiring ratings for product recommendation. In: ICEC ’07: Proceed-
ings of the ninth international conference on Electronic commerce, pp. 379–388. ACM, New
York, NY, USA (2007). DOI http://doi.acm.org/10.1145/1282100.1282171
31. Jin, R., Si, L.: A bayesian approach toward active learning for collaborative filtering. In: AUAI
’04: Proceedings of the 20th conference on Uncertainty in artificial intelligence, pp. 278–285.
AUAI Press, Arlington, Virginia, United States (2004)
32. Johar, M., Mookerjee, V., Sarkar, S.: Selling vs. profiling: Optimizing the offer set in web-
based personalization. Information Systems Research 25(2), 285–306 (2014).
33. John, R.C.S., Draper, N.R.: D-optimality for regression designs: A review. Technometrics
17(1), 15–23 (1975)
34. Kale, D., Liu, Y.: Accelerating active learning with transfer learning. In: Data Mining (ICDM),
2013 IEEE 13th International Conference on, pp. 1085–1090 (2013). DOI 10.1109/ICDM.
2013.160
35. Kapoor, A., Horvitz, E., Basu, S.: Selective supervision: Guiding supervised learning with
decision-theoretic active learning. In: Proceedings of International Joint Conference on Arti-
ficial Intelligence (IJCAI), pp. 877–882 (2007)
36. Karimi, R., Freudenthaler, C., Nanopoulos, A., Schmidt-Thieme, L.: Exploiting the char-
acteristics of matrix factorization for active learning in recommender systems. In: Pro-
ceedings of the Sixth ACM Conference on Recommender Systems, RecSys ’12, pp. 317–
320. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2366031. URL http:
//doi.acm.org/10.1145/2365952.2366031
37. Kohrs, A., Merialdo, B.: Improving collaborative filtering for new users by smart object selec-
tion. In: Proceedings of International Conference on Media Features (ICMF) (2001)
38. Le, Q.T., Tu, M.P.: Active learning for co-clustering based collaborative filtering. In: Com-
puting and Communication Technologies, Research, Innovation, and Vision for the Future
(RIVF), 2010 IEEE RIVF International Conference on, pp. 1–4 (2010). DOI 10.1109/RIVF.
2010.5633245
39. Leino, J., R¨
aih¨
a, K.J.: Case amazon: ratings and reviews as part of recommendations. In:
RecSys ’07: Proceedings of the 2007 ACM conference on Recommender systems, pp. 137–
140. ACM, New York, NY, USA (2007). DOI http://doi.acm.org/10.1145/1297231.1297255
40. Lomasky, R., Brodley, C., Aernecke, M., Walt, D., Friedl, M.: Active class selection. In: In
Proceedings of the European Conference on Machine Learning (ECML). Springer (2007)
41. McCallum, A., Nigam, K.: Employing em and pool-based active learning for text classifi-
cation. In: ICML ’98: Proceedings of the Fifteenth International Conference on Machine
Learning, pp. 350–358. San Francisco, CA, USA (1998)
42. Mcginty, L., Smyth, B.: On the Role of Diversity in Conversational Recommender Systems.
Case-Based Reasoning Research and Development pp. 276–290 (2003)

24 Active Learning in Recommender Systems 855
43. McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics
have hurt recommender systems. In: CHI ’06: CHI ’06 extended abstracts on Human factors
in computing systems, pp. 1097–1101. ACM Press, New York, NY, USA (2006). DOI http:
//doi.acm.org/10.1145/1125451.1125659
44. Nakamura, A., Abe, N.: Collaborative filtering using weighted majority prediction algorithms.
In: ICML ’98: Proceedings of the Fifteenth International Conference on Machine Learning,
pp. 395–403. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1998)
45. Pu, P., Chen, L.: User-Involved Preference Elicitation for Product Search and Recommender
Systems. AI magazine pp. 93–103 (2009). URL http://www.aaai.org/ojs/index.
php/aimagazine/article/viewArticle/2200
46. Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting
to know you: learning new user preferences in recommender systems. In: IUI ’02: Proceedings
of the 7th international conference on Intelligent user interfaces, pp. 127–134. ACM Press,
New York, NY, USA (2002). DOI http://doi.acm.org/10.1145/502716.502737
47. Rashid, A.M., Karypis, G., Riedl, J.: Influence in ratings-based recommender systems: An
algorithm-independent approach. In: SIAM International Conference on Data Mining, pp.
556–560 (2005)
48. Resnick, P., Sami, R.: The influence limiter: provably manipulation-resistant recommender
systems. In: Proceedings of the 2007 ACM conference on Recommender systems, RecSys
’07, pp. 25–32. ACM, New York, NY, USA (2007). DOI http://doi.acm.org/10.1145/1297231.
1297236. URL http://doi.acm.org/10.1145/1297231.1297236
49. Ricci, F., Nguyen, Q.N.: Acquiring and revising preferences in a critique-based mobile rec-
ommender system. IEEE Intelligent Systems 22(3), 22–29 (2007). DOI http://dx.doi.org/10.
1109/MIS.2007.43
50. Rokach, L., Naamani, L., Shmilovici, A.: Pessimistic cost-sensitive active learning of decision
trees for profit maximizing targeting campaigns. Data Mining and Knowledge Discovery
17(2), 283–316 (2008). DOI http://dx.doi.org/10.1007/s10618-008-0105-2
51. Roy, N., Mccallum, A.: Toward optimal active learning through sampling estimation of error
reduction. In: In Proc. 18th International Conf. on Machine Learning, pp. 441–448. Morgan
Kaufmann (2001)
52. Rubens, N., Sugiyama, M.: Influence-based collaborative active learning. In: Proceedings of
the 2007 ACM conference on Recommender systems (RecSys 2007). ACM (2007). DOI
http://doi.acm.org/10.1145/1297231.1297257
53. Rubens, N., Tomioka, R., Sugiyama, M.: Output divergence criterion for active learning in
collaborative settings. IPSJ Transactions on Mathematical Modeling and Its Applications
2(3), 87–96 (2009)
54. Saar-Tsechansky, M., Provost, F.: Decision-centric active learning of binary-outcome models.
Information Systems Research 18(1), 4–22 (2007). DOI http://dx.doi.org/10.1287/isre.1070.
0111
55. Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start
recommendations. In: SIGIR ’02: Proceedings of the 25th annual international ACM SIGIR
conference on Research and development in information retrieval, pp. 253–260. ACM, New
York, NY, USA (2002). DOI http://doi.acm.org/10.1145/564376.564421
56. Schohn, G., Cohn, D.: Less is more: Active learning with support vector machines. In: Proc.
17th International Conf. on Machine Learning, pp. 839–846. Morgan Kaufmann, San Fran-
cisco, CA (2000). URL citeseer.ist.psu.edu/schohn00less.html
57. Settles, B.: Active learning literature survey. Computer Sciences Technical Report 1648, Uni-
versity of Wisconsin–Madison (2009)
58. Settles, B., Craven, M.: An analysis of active learning strategies for sequence labeling tasks.
In: Proceedings of the Conference on Empirical Methods in Natural Language Processing
(EMNLP), pp. 1069–1078. ACL Press (2008)
59. Settles, B., Craven, M., Friedland, L.: Active learning with real annotation costs. In: Proceed-
ings of the NIPS Workshop on Cost-Sensitive Learning, pp. 1–10 (2008)

856 Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan
60. Settles, B., Craven, M., Ray, S.: Multiple-instance active learning. In: Advances in Neural
Information Processing Systems (NIPS), vol. 20, pp. 1289–1296. MIT Press (2008)
61. Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Computational Learning
Theory, pp. 287–294 (1992). URL citeseer.ist.psu.edu/seung92query.html
62. Sugiyama, M.: Active learning in approximately linear regression based on conditional expec-
tation of generalization error. Journal of Machine Learning Research 7, 141–166 (2006)
63. Sugiyama, M., Rubens, N.: A batch ensemble approach to active learning with model selec-
tion. Neural Netw. 21(9), 1278–1286 (2008). DOI http://dx.doi.org/10.1016/j.neunet.2008.
06.004
64. Sugiyama, M., Rubens, N., Mller, K.R.: Dataset Shift in Machine Learning, chap. A condi-
tional expectation approach to model selection and active learning under covariate shift. MIT
Press, Cambridge (2008)
65. Sutherland, D.J., P´
oczos, B., Schneider, J.: Active learning and search on low-rank matrices.
In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining, KDD ’13, pp. 212–220. ACM, New York, NY, USA (2013). DOI 10.1145/
2487575.2487627. URL http://doi.acm.org/10.1145/2487575.2487627
66. Swearingen, K., Sinha, R.: Beyond algorithms: An hci perspective on recommender sys-
tems. ACM SIGIR 2001 Workshop on Recommender Systems (2001). URL http:
//citeseer.ist.psu.edu/cache/papers/cs/31330/http:zSzzSzweb.
engr.oregonstate.eduzSz\˜{}herlockzSzrsw2001zSzfinalzSzfull\
_length\_paperszSz4\_swearingenzPz.pdf/swearingen01beyond.pdf
67. Tong, S., Koller, D.: Support vector machine active learning with applications to text clas-
sification. In: P. Langley (ed.) Proceedings of ICML-00, 17th International Conference on
Machine Learning, pp. 999–1006. Morgan Kaufmann Publishers, San Francisco, US, Stan-
ford, US (2000). URL citeseer.ist.psu.edu/article/tong01support.html
68. Yu, K., Bi, J., Tresp, V.: Active learning via transductive experimental design. In: Proceedings
of the 23rd Int. Conference on Machine Learning ICML ’06, pp. 1081–1088. ACM, New
York, NY, USA (2006). DOI http://doi.acm.org/10.1145/1143844.1143980
69. Zhao, L., Pan, S.J., Xiang, E.W., Zhong, E., Lu, Z., Yang, Q.: Active transfer learning for
cross-system recommendation. In: AAAI (2013)
70. Zhao, X., Zhang, W., Wang, J.: Interactive collaborative filtering. In: Proceedings of the
22nd ACM international conference on Conference on information &#38; knowledge man-
agement, CIKM ’13, pp. 1411–1420. ACM, New York, NY, USA (2013). DOI 10.1145/
2505515.2505690. URL http://doi.acm.org/10.1145/2505515.2505690