No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
The ‘Unreasonable’ Effectiveness of Graphical User Interfaces for Recommender Systems
... Over the past four decades, two primary approaches have emerged to advance Responsible AI [6,9], each representing prolific research domains. The first is a contextual approach involving regulation through legal frameworks (see Fig. 1). ...
... Vital as it may be, however, the interactional approach to Responsible AI has been underemphasized and requires rapid development [9,14,15]. Active involvement from professionals within the design community is essential for the advancement of Responsible AI. To facilitate such effective contribution, designers require resources that aid in shaping human-AI interactions [15]. ...
... In a study amongst 200 practitioners worldwide, Smits & Van Turnhout established professionals' need for such research and for other resources to help them navigate the still new design challenges associated with algorithmic affordances [14]. However, while studies on algorithms and datasets abound [9,42,43] and while AI is identified as a key technology, the studies on design of the algorithmic affordances are falling behind in volume and attention [9,18,42,43]. ...
In this paper, we argue that the creation of Responsible AI over the past four decades has predominantly relied on two approaches: contextual and technical. While both are indispensable, we contend that a third equally vital approach , focusing on human-AI interaction design, has been relatively neglected, despite the ongoing efforts of pioneers in the field. Through the presentation of four case studies of real-world AI systems, we illustrate, however, how small design choices impact an AI platform's responsibleness independent of the technical or contextual level. We advocate, therefore, for a larger role for design in the creation of Responsible AI, and we call for increased research efforts in this area to advance our understanding of human-AI interaction design in Responsible AI. This includes both smaller case studies, such as those presented in this paper, that replicate or swiftly test earlier findings in different contexts, as well as larger, more comprehensive studies that lay the foundations for a framework.
... To accommodate for such a significant design shift, practitioners require access to all available resources, including current, relevant research presented in a manner that resonates with their professional needs. While studies on the accuracy, efficiency, and effectiveness of recommender algorithms abound for engineers working on the backend of the system [2,3,[11][12][13][14], the same cannot be said for professionals working on the front end of screen-based recommenders, as the number of studies in this area is limited [2][3][4]13]. Even lower is the number of studies that find their way into design practice [15,16]. ...
... To accommodate for such a significant design shift, practitioners require access to all available resources, including current, relevant research presented in a manner that resonates with their professional needs. While studies on the accuracy, efficiency, and effectiveness of recommender algorithms abound for engineers working on the backend of the system [2,3,[11][12][13][14], the same cannot be said for professionals working on the front end of screen-based recommenders, as the number of studies in this area is limited [2][3][4]13]. Even lower is the number of studies that find their way into design practice [15,16]. ...
... One result of this shift was that recommender system evaluation frameworks also started to incorporate user-centric criteria [11,12,36,37]. Despite the growing emphasis on user experience, however, the research into these user-centric design qualities still very much remained the domain of the data scientist [2,38,39]. The research included questions such as 'how to adapt algorithmic calculations to generate a still accurate yet novel and diverse list of recommendations' (for the design qualities novelty and diversity) [40,41] or 'how to extract explanations from complex algorithms' (for transparency) [35]. ...
Brian Shackel Award winner for most outstanding contribution with international impact in the field of human interaction with, and human use of, computers and information technology
Reviewers' Choice winner
The design of recommender systems’ graphical user interfaces (GUIs) is critical for a user's experience with these systems. However, most research into recommenders focuses on algorithms, overlooking the design of their interfaces. Additionally, the studies on the design of recommender interfaces that do exist do not always manage to cross the research-practice gap. This disconnect may be due to a lack of alignment between academic focus and the most pressing needs of practitioners, as well as the way research findings are communicated. To address these issues, this paper presents the results of a comprehensive study involving 215 designers worldwide, aiming to identify the primary challenges in designing recommender GUIs and the resources practitioners need to tackle those challenges. Building on these findings, this paper proposes a practice-led research agenda for the human-computer interaction community on designing recommender interfaces and suggestions for more accessible and actionable ways of disseminating research results in this domain. KeywordsRecommender systemInterface designResearch-practice gapAlgorithmic affordances
... Unfortunately, many datasets have sparsity issues [12], a lack of meaningful content diversity [11,13], or omit descriptions of users' backgrounds [14,15,11]. Furthermore, while previous studies found that the image as well as the visualization/display style of a news article are crucial for predicting user engagement [16,17], such information is not present in any of the existing datasets. To counteract and alleviate these weaknesses, we present the Informfully Dataset with Enhanced Attributes (IDEA). 1 Table 1 Overview of the attributes available in open-source datasets in the news domain: user data on session activity, survey data for background information on users, text and image of news articles, user-article interactions, like/dislike ratings, recommendation lists, and reference screenshots for article presentation. ...
... For each article, the collection holds the title, lead, and accompanying metadata, such as the publication outlet, author, and image URL. 13 Figure 3 presents an overview of the quantitative data on daily active users ( Figure 3A), daily user-item interactions ( Figure 3B), distribution of news topics among the opened and read articles ( Figure 3C), and article length ( Figure 3D). 16 We see almost constant daily active users counts around 400 (M = 388.93, SD = 77.62) ...
In this paper, we present the Informfully Dataset with Enhanced Attributes (IDEA) for news article recommendations. The dataset consists of an open-source collection of user profiles, news articles with a high topic and outlet diversity, item recommendations, and rich user-item interactions from a field study on behavioral changes in news consumption. The records include both quantitative data from real-time session tracking as well as self-reported data from user surveys on satisfaction with news, knowledge acquisition, and personal background information. This paper outlines the data collection procedure and potential use cases of the dataset for designing normative recommender systems. It provides the documentation of all data collections together with insights into the data quality.
... Offline evaluations can also fall short of predicting real-world user interactions, as not only do the opinions and preferences of the users matter, but also how the items are presented [28]. The presentation and the display style of recommendations can have a significant impact on the selection made by users [2,3] and remains an open problem [7]. ...
... Informfully (front and back end) is built with React Native, 2 Meteor, 3 and MongoDB. 4 During the design and development phase, we deliberately tried to have as few dependencies (in the form of overlap and communication) between these three components. ...
This paper presents Informfully, a research platform for content distribution and user studies. Informfully allows to push algorithmically curated text, image, audio, and video content to users and automatically generates a detailed log of their consumption history. As such, it serves as an open-source platform for conducting user experiments to investigate the impact of item recommendations on users' consumption behavior. The platform was designed to accommodate different experiment types through versatility, ease of use, and scalability. It features three core components: 1) a front end for displaying and interacting with recommended items, 2) a back end for researchers to create and maintain user experiments, and 3) a simple JSON-based exchange format for ranked item recommendations to interface with third-party frameworks. We provide a system overview and outline the three core components of the platform. A sample workflow is shown for conducting field studies incorporating multiple user groups, personalizing recommendations , and measuring the effect of algorithms on user engagement. We present evidence for the versatility, ease of use, and scalability of Informfully by showcasing previous studies that used our platform.
... Ultimately, how the user perceives the tool's support determines whether they will choose to adopt, ignore, or negotiate the AI's recommendations [5][6][7]. Initially, studies on these AI tools were technical in nature and focused on qualities such as accuracy and efficiency and how various algorithmic models affected those qualities [8]. It was only at a later stage that the user experience was considered part of the equation [9,10], and studies started to examine the impact of recommender systems on other qualities of the user experience, such as novelty, diversity, fairness, safety and autonomy [5,11,12]. ...
... The impact of algorithmic affordances on interaction qualities, such as controllability, autonomy, transparency, interpretability, autonomy, fun, etc. has been demonstrated overwhelmingly in literature [3,8,11,20,[26][27][28][29][30][31][32][33][34]. However, currently, no systematic overview exists in which the individual studies and their impact on the various interaction qualities have been brought together. ...
The user's experience with a recommender system is significantly shaped by the dynamics of user-algorithm interactions. These interactions are often evaluated using interaction qualities, such as controllability, trust, and autonomy , to gauge their impact. As part of our effort to systematically categorize these evaluations, we explored the suitability of the interaction qualities framework as proposed by Lenz, Dieffenbach and Hassenzahl. During this examination, we uncovered four challenges within the framework itself, and an additional external challenge. In studies examining the interaction between user control options and interaction qualities, interdependencies between concepts, inconsistent terminology , and the entity perspective (is it a user's trust or a system's trustworthiness) often hinder a systematic inventory of the findings. Additionally, our discussion underscored the crucial role of the decision context in evaluating the relation of algorithmic affordances and interaction qualities. We propose dimensions of decision contexts (such as 'reversibility of the decision', or 'time pressure'). They could aid in establishing a systematic three-way relationship between context attributes , attributes of user control mechanisms, and experiential goals, and as such they warrant further research. In sum, while interaction qualities framework serves as a foundational structure for organizing research on evaluating the impact of algorithmic affordances, challenges related to interdependencies and context specific influences remain. These challenges necessitate further investigation and subsequent refinement and expansion of the framework.
... A user interface that clearly conveys the intent behind the recommended items, such as "users like you also liked this" or "personalized for you, " can significantly boost the system's effectiveness. On the other hand, a cluttered or confusing interface can cause user frustration and a decrease in overall system performance [3]. For instance, a system that recommends a movie and simply displays the movie's title and a generic image without any additional context may not be as effective as one that shows the movie's genre, a brief summary, and the number of users who have liked similar movies. ...
... Unlike the AI core problem, which can be solved almost purely mathematically, the UI problem mainly involves soft human factors that are difficult to predict [113]. Previous studies have tested several factors influencing the adoption of AI recommendations and human-AI trust. ...
Information sharing (IS) occurs in almost every action daily. IS holds benefits for its users, but it is also a source of privacy violations and costs. Human users struggle to balance this trade-off. This reality calls for Artificial Intelligence (AI)-based agent assistance that surpasses humans’ bottom-line utility, as shown in previous research. However, convincing an individual to follow an AI agent’s recommendation is not trivial; therefore, this research's goal is establishing trust in machines. Based on the Design of Experiments (DOE) approach, we developed a methodology that optimizes the user interface (UI) with a target function of maximizing the acceptance of the AI agent's recommendation. To empirically demonstrate our methodology, we conducted an experiment with eight UI factors and n = 64 human participants, acting in a Facebook simulator environment, and accompanied by an AI agent assistant. We show how the methodology can be applied to enhance AI agent user acceptance on IS platforms by selecting the proper UI. Additionally, due to its versatility, this approach has the potential to optimize user acceptance in multiple domains as well.
... As the relevance of recommender systems increases, so does the need for research into user-centred design and development. Furthermore, recommendations need to be provided to the user in a way that gives them the best experience [10,18,36,45]. As a result, user feedback plays an important role in the design and development process of the recommender system. ...
The integration of digital technologies in education has led to major changes, including the rise of project-based work. However, learners often face challenges in navigating through the abundance of available resources. To tackle this issue, a recommender system is being designed to support a digital project portfolio platform. The system aims to improve transparency, networking, collaboration, and cooperation among educational stakeholders. Personalised recommendations are crucial to match resources to individual learners’ needs, improve their experience and foster potential collaborations and entrepreneurial ideas. To achieve this goal, understanding the learner’s requirements is essential. This study aims to investigate the needs of students in the development of a recommendation system for a project portfolio platform in education through an extensive literature review and qualitative semi-structured interviews. The study involved interviews with seven students from different study programs and educational departments using pre-designed mockups and presentation slides. The results showed that students considered the use of a user-centred recommendation system for a project portfolio platform in educational institutions to be important and valuable. Further research could include other study programs in the qualitative survey. The representativeness of the overall findings can then be evaluated using quantitative methods.
... For many years the evaluation of AI has been primarily technically oriented [12,13]. The success of a recommender systems, for instance, was based on the accuracy and efficiency of the algorithms. ...
Algorithmic affordances are defined as user interaction mechanisms that allow users tangible control over AI algorithms, such as recommender systems. Designing such algorithmic affordances, including assessing their impact, is not straightforward and practitioners state that they lack resources to design adequately for interfaces of AI systems. This could be amended by creating a comprehensive pattern library of algorithmic affordances. This library should provide easy access to patterns, supported by live examples and research on their experiential impact and limitations of use. The Algorithmic Affordances in Rec-ommender Interfaces workshop aimed to address key challenges related to building such a pattern library, including pattern identification features, a framework for systematic impact evaluation, and understanding the interaction between al-gorithmic affordances and their context of use, especially in education or with users with a low algorithmic literacy. Preliminary solutions were proposed for these challenges.
... In some cases, the challenge of convincing the user to adopt the recommendations may even be more challenging than calculating the best action (the AI core). Unlike the AI core problem, which can be solved almost purely mathematically, the UI problem mainly involves soft human factors which are difficult to predict (Beel & Dixon, 2021). To address this problem, we proposed a methodology in the form of an experimental design. ...
Information-sharing (IS) occurs in almost every action of our daily life. IS holds benefits for its users, but it is also a source of privacy violations. Human users struggle to balance this trade-off between the potential benefits and the resulting costs. This reality calls for Artificial-Intelligence (AI)-based agent assistance that surpasses humans’ bottom-line utility, as shown in previous research. However, convincing an individual to follow an AI agent’s recommendation is not trivial; therefore, the current research goal is establishing trust in the machine. To this end, based on the Design of Experiments (DOE) approach, we developed a methodology that optimizes the user-interface (UI) with a target function of maximizing the AI agent recommendation acceptance. To empirically demonstrate our methodology, we conducted an experiment with eight UI factors and (n=64) human participants acting in a Facebook simulator environment accompanied by an AI-agent assistant. Based on the results, we showed how the methodology can be implemented to optimize the agent’s users’ acceptance. Finally, while our methodology was tested empirically on an IS platform, it could be applied straightforwardly in other domains.
... However, the presentation of recommendations has not received the same attention as other more user-oriented aspects. Very few authors have explored alternatives to one-dimensional lists, presenting items and arranging the user interface in different ways (Lousame and Sánchez, 2009;Nanou et al., 2010;Guntuku et al., 2016;Beel and Dixon, 2021). This seems inexplicable given the potentially strong impact of the presentation format on the user Loepp . ...
For a long time, recommender systems presented their results in the form of simple item lists. In recent years, however, multi-list interfaces have become the de-facto standard in industry, presenting users with numerous collections of recommendations, one below the other, each containing items with common characteristics. Netflix's interface, for instance, shows movies from certain genres, new releases, and lists of curated content. Spotify recommends new songs and albums, podcasts on specific topics, and what similar users are listening to. Despite their popularity, research on these so-called “carousels” is still limited. Few authors have investigated how to simulate the user behavior and how to optimize the recommendation process accordingly. The number of studies involving users is even smaller, with sometimes conflicting results. Consequently, little is known about how to design carousel-based interfaces for achieving the best user experience. This mini review aims to organize the existing knowledge and outlines directions that may improve the multi-list presentation of recommendations in the future.
... The design decisions involved in presenting explanations can unintentionally introduce further bias into AI tools. For instance, the manner in which an explanation is visualized or presented to the user could influence their decision-making process [4,10]. This could lead to a biased text outcome, as recent work [25] suggests that collaborative writing with AI affects users' views, possibly affecting the contents of the written text. ...
We highlight the challenges faced by non-native speakers when using AI writing assistants to paraphrase text. Through an interview study with 15 non-native English speakers (NNESs) with varying levels of English proficiency, we observe that they face difficulties in assessing paraphrased texts generated by AI writing assistants, largely due to the lack of explanations accompanying the suggested paraphrases. Furthermore, we examine their strategies to assess AI-generated texts in the absence of such explanations. Drawing on the needs of NNESs identified in our interview, we propose four potential user interfaces to enhance the writing experience of NNESs using AI writing assistants. The proposed designs focus on incorporating explanations to better support NNESs in understanding and evaluating the AI-generated paraphrasing suggestions.
... These issues are recently addressed by many studies where adaptive interfaces [11,18,22] have been developed. Therefore, a mechanism to incorporate User Experience (UX) for embedded customization in the user interface (UI) is required in order to guarantee a longer adaptation of the system and a great satisfaction of the final user [1,5]. Among adaptive interfaces, conversational recommender systems [3,12,13] play a crucial role. ...
... Yet, self-reinforcing loops, constraining the user to certain interaction mechanisms, must be avoided [46]. For this reason, among others, it is finally important to explore the possibilities for the 2d) presentation: Earlier works on RS have shown, e.g., significant effects of presenting items or the entire interface in different ways [4,19,40,44]. Whereas only behavioral data were considered in these cases, studying factors such as personality has a long tradition in user interface design [3]. This might turn out useful for an adaptive presentation of DA, especially for raising awareness of the mechanisms the system has predicted to be of relevance before, in a persuasive but unobtrusive manner: Explanations, currently used in RS mainly to explain item recommendations [55], but also perceived differently depending on user characteristics [22], could be used, e.g., to highlight the benefits of continuing the interaction with a specific DA. ...
On the one hand, users' decision making in today's web is supported in numerous ways, with mechanisms ranging from manual search over automated recommendation to intelligent advisors. The focus on algorithmic accuracy, however, is questioned more and more. On the other hand, although the boundaries between the mechanisms are blurred increasingly, research on user-related aspects is still conducted separately in each area. In this position paper, we present a research agenda for providing a more holistic solution, in which users are supported with the right decision aid at the right time depending on personal characteristics and situational needs.
Every entity must communicate seamlessly in an interconnected network. Personalised delivery of information in an open private network(s) among consumers should be transparent while maintaining privacy. In information‐centric content delivery networks (ICDN) or data‐centric content delivery networks (DCDN), data delivery relies heavily on sharing data with third‐party entities, resulting in a privacy breach. To overcome these challenges, this article proposes a novel communication model with a consumer‐centric content delivery network (CCDN) approach for an internet‐of‐things (IoT) network that deals in personalised information delivery with users' identity and location privacy. The article derives motivation from a use case‐based analysis of the proposed communication model (CCDN) by incorporating the latest technologies like software‐defined networks (SDN) and content delivery network (CDN) into IoT Networks to achieve a higher level of personalisation while maintaining user privacy. CCDN leverages SDN‐based CDN for IoT networks in the designed use case that strictly adheres to user‐defined preferences/demands. The article suggests an environment with a possible topology implementation and describes the elements that can help achieve it. The article investigates the need for personalisation and how it is contrary to maintaining user privacy. The article also contributes a survey of the underlying technologies while linking them to IoT.
Recommendations-as-a-Service (RaaS) ease the process for small and medium-sized enterprises (SMEs) to offer product recommendations to their customers. Current RaaS, however, suffer from a one-size-fits-all concept, i.e. they apply the same recommendation algorithm for all SMEs. We introduce Darwin & Goliath, a RaaS that features multiple recommendation frameworks (Apache Lucene, TensorFlow, ...), and identifies the ideal algorithm for each SME automatically. Darwin & Goliath further offers per-instance algorithm selection and a white label feature that allows SMEs to offer a RaaS under their own brand. Since November 2018, Darwin & Goliath has delivered more than 1m recommendations with a CTR = 0.5%.
Choice overload describes a situation in which a person has difficulty in making decisions due to too many options. We examine choice overload when displaying related-article recommendations in digital libraries, and examine the effectiveness of recommendation algorithms in this domain. We first analyzed existing digital libraries, and found that only 30% of digital libraries show related-article recommendations to their users. Of these libraries, the majority (74%) displays 3–5 related articles; 28% of them display 6–10 related articles; and no digital library displayed more than ten related-article recommendations. We then conducted our experimental evaluation through GESIS’ digital library Sowiport, with recommendations delivered by recommendations-as-a-service provider Mr. DLib. We use four metrics to analyze 41.3 million delivered recommendations: click-through rate (CTR), percentage of clicked recommendation sets (clicked set rate, CSR), average clicks per clicked recommendation set (ACCS), and time to first click (TTFC), which is the time between delivery of a set of recommendations to the first click. These metrics help us to analyze choice overload and can yield evidence for finding the ideal number of recommendations to display. We found that with increasing recommendation set size, i.e., the numbers of displayed recommendations, CTR decreases from 0.41% for one recommendation to 0.09% for 15 recommendations. Most recommendation sets only receive one click. ACCS increases with set size, but increases more slowly for six recommendations and more. When displaying 15 recommendations, the average clicks per set is at a maximum (1.15). Similarly, TTFC increases with larger recommendation set size, but increases more slowly for sets of more than five recommendations. While CTR and CSR do not indicate choice overload, ACCS and TTFC point toward 5–6 recommendations as being optimal for Sowiport. Content-based filtering yields the highest CTR with 0.118%, while stereotype recommendations yield the highest ACCS (1.28). Stereotype recommendations also yield the highest TTFC. This means that users take more time before clicking stereotype recommendations when compared to recommendations based on other algorithms.
Facilitating serendipitous recommendations via a user interface for a recommender system is challenging. Recommender systems that facilitate serendipitous recommendations create what are considered to be beyond-accuracy experiences. Because serendipitous recommendations are unexpected, users may experience some trust issues with the recommender system. This paper reports on the use of transparency for connection-making (a process/stage of serendipitous experience) in serendipity recommendations generation and examines trust with respect to an interactive serendipitous recommender system. We developed a user interface for JabRef (related work tab). The experiment studied the responses of users, comprised of undergraduate students, in a university environment. Data was collected for both subjective and objective evaluations. The analysis was used to determine the significance and impact of transparency in recommender systems in which serendipitous recommendations are generated. The results showed that greater transparency has a positive impact on user experience, helps with connection-making, and enhances the trust of learners who use recommender systems for learning purposes. Such systems can be used by faculty members and students during research and supervision in order to facilitate finding new research topics and trust in such systems
This paper presents a description of integration of the Mr. DLib scientific recommender system into the JabRef reference manager. Scientific recommender systems help users identify relevant papers out of vast amounts of existing literature. They are particularly useful when used in combination with reference managers. Over 85% of JabRef users stated that they would appreciate the integration of a recommender system. However, the implementation of literature recommender systems requires experience and resources that small companies cannot afford. With the desires of users in mind, we integrated the Mr. DLib scientific recommender system into JabRef. Using Mr. DLib's recommendations-as-a-service, JabRef users can find relevant literature and keep themselves informed about the state of the art in their respective fields.
Recommending content to users involves understanding a) what to present and b) how to present them, so as to increase quality of experience (QoE) and thereby, content consumption. This work attempts to address the question of how to present contents in a way so that the user finds it easy to get to desired content. While the process of User Interface (UI) design is dependent on several human factors, there are basic design components and their combination that have to be common to any recommender system user interface. Personalization of the UI design process involves picking the right components and their combination, and presenting a UI to suit the usage behavior of an individual user, so as to enhance the QoE. This work proposes a system that learns from a user's content consumption patterns and makes some recommendations regarding how to present the content for the user (in the context of Video-On-Demand/Live-TV services on Computer displays), so as to enhance the QoE of the recommender system.
How do click-through rates vary between research paper recommendations previously shown to the same users and recommendations shown for the very first time? To answer this question we analyzed 31,942 research paper recommendations given to 1,155 students and researchers with the literature management software Docear. Results indicate that recommendations should only be given once Click-through rates for ‘fresh’, i.e. previously unknown, recommendations are twice as high as for already known recommendations. Results also show that some users are ‘oblivious’. It frequently happened that users clicked on recommendations they already knew. In one case the same recommendation was shown six times to the same user and the user clicked on it each time again. Overall, around 50% of clicks on reshown recommendations were such ‘oblivious-clicks’.
Mind-maps have been widely neglected by the information retrieval (IR) community. However, there are an estimated two million active mind-map users, who create 5 million mind-maps every year, of which a total of 300,000 is publicly available. We believe this to be a rich source for information retrieval applications, and present eight ideas on how mind-maps could be utilized by them. For instance, mind-maps could be utilized to generate user models for recommender systems or expert search, or to calculate relatedness of web-pages that are linked in mind-maps. We evaluated the feasibility of the eight ideas, based on estimates of the number of available mind-maps, an analysis of the content of mind-maps, and an evaluation of the users’ acceptance of the ideas. We concluded that user modelling is the most promising application with respect to mind-maps. A user modelling prototype – a recommender system for the users of our mind-mapping software Docear – was implemented, and evaluated. Depending on the applied user modelling approaches, the effectiveness, i.e. click-through rate on recommendations, varied between 0.28% and 6.24%. This indicates that mind-map based user modelling is promising, but not trivial, and that further research is required to increase effectiveness.
Research on recommender systems has traditionally focused on the development of algorithms to improve accuracy of recommendations. So far, little research has been done to enable user interaction with such systems as a basis to support exploration and control by end users. In this paper, we present our research on the use of information visualization techniques to interact with recommender systems. We investigated how information visualization can improve user understanding of the typically black-box rationale behind recommendations in order to increase their perceived relevance and meaning and to support exploration and user involvement in the recommendation process. Our study has been performed using TalkExplorer, an interactive visualization tool developed for attendees of academic conferences. The results of user studies performed at two conferences allowed us to obtain interesting insights to enhance user interfaces that integrate recommendation technology. More specifically, effectiveness and probability of item selection both increase when users are able to explore and interrelate multiple entities -- i.e. items bookmarked by users, recommendations and tags.
Research on recommender systems typically focuses on the accuracy of prediction algorithms. Because accuracy only partially constitutes the user experience of a recommender system, this paper proposes a framework that takes a user-centric approach to recommender system evaluation. The framework links objective system aspects to objective user behavior through a series of perceptual and evaluative constructs (called subjective system aspects and experience, respectively). Furthermore, it incorporates the influence of personal and situational characteristics on the user experience. This paper reviews how current literature maps to the framework and identifies several gaps in existing work. Consequently, the framework is validated with four field trials and two controlled experiments and analyzed using Structural Equation Modeling. The results of these studies show that subjective system aspects and experience variables are invaluable in explaining why and how the user experience of recommender systems comes about. In all studies we observe that perceptions of recommendation quality and/or variety are important mediators in predicting the effects of objective system aspects on the three components of user experience: process (e.g. perceived effort, difficulty), system (e.g. perceived system effectiveness) and outcome (e.g. choice satisfaction). Furthermore, we find that these subjective aspects have strong and sometimes interesting behavioral correlates (e.g. reduced browsing indicates higher system effectiveness). They also show several tradeoffs between system aspects and personal and situational characteristics (e.g. the amount of preference feedback users provide is a tradeoff between perceived system usefulness and privacy concerns). These results, as well as the validated framework itself, provide a platform for future research on the user-centric evaluation of recommender systems.
Recommender Systems have been developed to help people make choices, for instance when deciding what books to buy or movies
to see. Research to date has focused on developing algorithms to improve the predictive accuracy of recommender systems. This
paper presents an HCI approach to recommender systems design, based on the strategies people employ when seeking advice in
taste domains from various sources. The results from a qualitative study with 44 participants show that participants have
different requirements for different choice domains. In taste domains, the relationship between the advice seeker and recommender
is extremely important, so ways of indicating social closeness and taste overlap are required. Recommender systems must establish
a connection between the advice seeker and recommenders through explanation interfaces and communication functions.
At Brown University, there is excitement of having access to the Brown Corpus, containing one million English words. Since then, we have seen several notable corpora that are about 100 times larger, and in 2006, Google released a trillion-word corpus with frequency counts for all sequences up to five words long. In some ways this corpus is a step backwards from the Brown Corpus: it's taken from unfiltered Web pages and thus contains incomplete sentences, spelling errors, grammatical errors, and all sorts of other errors. It's not annotated with carefully hand-corrected part-of-speech tags. But the fact that it's a million times larger than the Brown Corpus outweighs these drawbacks. A trillion-word corpus - along with other Web-derived corpora of millions, billions, or trillions of links, videos, images, tables, and user interactions - captures even very rare aspects of human behavior. So, this corpus could serve as the basis of a complete model for certain tasks - if only we knew how to extract the model from the data.
A recommender system’s ability to establish trust with users and convince them of its recommendations, such as which camera or PC to purchase, is a crucial design factor especially for e-commerce environments. This observation led us to build a trust model for recommender agents with a focus on the agent’s trustworthiness as derived from the user’s perception of its competence and especially its ability to explain the recommended results. We present in this article new results of our work in developing design principles and algorithms for constructing explanation interfaces. We show the effectiveness of these principles via a significant-scale user study in which we compared an interface developed based on these principles with a traditional one. The new interface, called the organization interface where results are grouped according to their tradeoff properties, is shown to be significantly more effective in building user trust than the traditional approach. Users perceive it more capable and efficient in assisting them to make decisions, and they are more likely to return to the interface. We therefore recommend designers to build trust-inspiring interfaces due to their high likelihood to increase users’ intention to save cognitive effort and the intention to return to the recommender system.
In a world where the number of choices can be overwhelming, recommender systems help users find and evaluate items of interest. They do so by connecting users with information regarding the content of recommended items or the opinions of other individuals. Such systems have become powerful tools in domains such as electronic commerce, digital libraries, and knowledge management. In this paper, we address such systems and introduce a new class of recommender system called meta-recommenders. Meta-recommenders provide users with personalized control over the generation of a single recommendation list formed from a combination of rich data using multiple information sources and recommendation techniques. We discuss experiments conducted to aid in the design of interfaces for a meta-recommender in the domain of movies. We demonstrate that meta-recommendations fill a gap in the current design of recommender systems. Finally, we consider the challenges of building real-world, usable meta-recommenders across a variety of domains.
This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.
With the retail electronic commerce being a major global shopping phenomenon, retailers need to develop additional tools to improve their sales. One such tool is a recommender system through which the shopping page recommends products to the shoppers using their past Web shopping and product search behaviour. While recommender systems are common, few studies exist regarding their usability and user preferences. In this study, a structured survey concerning what recommender systems should contain and how this content should be presented was administered on 131 college-aged online shoppers. Results indicate participants prefer specific recommender content. Price, image and names of products are identified as essential information, while product promotions, customer ratings and feedback are identified as secondary types of information. Shoppers preferred short and relevant recommender information, with a maximum of three recommendations on one page. Future studies may explore differences in preference of recommender systems based on different product types.
Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.
Recommender systems use people's opinions about items in an information domain to help people choose other items. These systems have succeeded in domains as diverse as movies, news articles, Web pages, and wines. The psychological literature on conformity suggests that in the course of helping people make choices, these systems probably affect users' opinions of the items. If opinions are influenced by recommendations, they might be less valuable for making recommendations for other users. Further, manipulators who seek to make the system generate artificially high or low recommendations might benefit if their efforts influence users to change the opinions they contribute to the recommender. We study two aspects of recommender system interfaces that may affect users' opinions: the rating scale and the display of predictions at the time users rate items. We find that users rate fairly consistently across rating scales. Users can be manipulated, though, tending to rate toward the prediction the system shows, whether the prediction is accurate or not. However, users can detect systems that manipulate predictions. We discuss how designers of recommender systems might react to these findings.
Automated collaborative filtering (ACF) systems predict a person's affinity for items or information by connecting that person's recorded interests with the recorded interests of a community of people and sharing ratings between likeminded persons. However, current recommender systems are black boxes, providing no transparency into the working of the recommendation. Explanations provide that transparency, exposing the reasoning and data behind a recommendation. In this paper, we address explanation interfaces for ACF systems -- how they should be implemented and why they should be implemented. To explore how, we present a model for explanations based on the user's conceptual model of the recommendation process. We then present experimental results demonstrating what components of an explanation are the most compelling. To address why, we present experimental evidence that shows that providing explanations can improve the acceptance of ACF systems. We also describe some initial explor...
On many e-commerce and media streaming sites, the user interface
(UI) consists of multiple lists of item suggestions. The items in each list are usually chosen based on pre-defined strategies and, e.g., show movies of the same genre or category. Such interfaces are common in practice, but there is almost no academic research regarding the optimal design and arrangement of such multi-list UIs for recommenders. In this paper, we report the results of an exploratory user study that examined the effects of various design alternatives on the decision-making behavior of users in the context of similar-item recommendations. Our investigations showed, among other aspects, that decision-making is slower and more demanding with multi-list interfaces, but that users also explore more options before making a decision. Regarding the selection of the algorithm to retrieve similar items, our study furthermore reveals the importance of considering social-based similarity measures.
The multi-armed bandit is an important framework for balancing exploration with exploitation in recommendation. Exploitation recommends content (e.g., products, movies, music playlists) with the highest predicted user engagement and has traditionally been the focus of recommender systems. Exploration recommends content with uncertain predicted user engagement for the purpose of gathering more information. The importance of exploration has been recognized in recent years, particularly in settings with new users, new items, non-stationary preferences and attributes. In parallel, explaining recommendations ("recsplanations") is crucial if users are to understand their recommendations. Existing work has looked at bandits and explanations independently. We provide the first method that combines both in a principled manner. In particular, our method is able to jointly (1) learn which explanations each user responds to; (2) learn the best content to recommend for each user; and (3) balance exploration with exploitation to deal with uncertainty. Experiments with historical log data and tests with live production traffic in a large-scale music recommendation service show a significant improvement in user engagement.
Automated recommendations have become a ubiquitous part of today’s online user experience. These systems point us to additional items to purchase in online shops, they make suggestions to us on movies to watch, or recommend us people to connect with on social websites. In many of today’s applications, however, the only way for users to interact with the system is to inspect the recommended items. Often, no mechanisms are implemented for users to give the system feedback on the recommendations or to explicitly specify preferences, which can limit the potential overall value of the system for its users.
Academic research in recommender systems is largely focused on algorithmic approaches for item selection and ranking. Nonetheless, over the years a variety of proposals were made on how to design more interactive recommenders. This work provides a comprehensive overview on the existing literature on user interaction aspects in recommender systems. We cover existing approaches for preference elicitation and result presentation, as well as proposals that consider recommendation as an interactive process. Throughout the work, we furthermore discuss examples of real-world systems and outline possible directions for future works.
Automated recommendations have become a common feature of modern online services and mobile apps. In many practical applications, the means provided for users to interact with recommender systems (e.g., to state explicit preferences or to provide feedback on the recommendations) are, however, very limited. In order to improve such systems and consequently user satisfaction, much research work has been done over the years to build richer and more intelligent user interfaces for recommender systems. In this tutorial, we provide a comprehensive overview of existing approaches to user interaction aspects of recommender systems, with a special focus on explanation interfaces. We also provide examples of real-world systems that implement advanced interaction mechanisms and discuss open challenges in the field.
Generating a useful recommendation is only the first step in creating a recommendation system. For the system to have value, the recommendations must be delivered with a user interface that allows the user to become aware that recommendations are available, to determine if any of the recommendations have value for them and to be able to act upon a recommendation. By synthesizing previous results from general recommendation system research and software engineering recommendation system research, we discuss the factors that affect whether or not a user considers and accepts recommendations generated by a system. These factors include the ease with which a recommendation can be understood and the level of trust a user assigns to a recommendation. In this chapter, we will describe these factors and the opportunities for future research towards helping getting the user interface of a recommendation system just right.
[from: ceur-ws.org/Vol-1050/Decisions2013Proceedings.pdf] Users interact with recommender systems to obtain useful information about products or services that may be of interest for them. But, while users are interacting with a recommender system to fulfill a primary task, which is usually the selection of one or more items, they are facing several other decision problems. For instance, they may be requested to select specific feature values (e.g., camera’s size, zoom) as criteria for a search, or they could have to identify features to be used in a critiquing based recommendation session, or they may need to select a repair proposal for inconsistent user preferences when interacting with a recommender. In all these scenarios, and in many others, users of recommender systems are facing decision tasks.
The complexity of decision tasks, limited cognitive resources of users, and the tendency to keep the overall decision effort as low as possible is modeled by theories that conjecture “bounded rationality”, i.e., users are exploiting decision heuristics rather than trying to take an optimal. Furthermore, preferences of users will likely change throughout a recommendation session, i.e., preferences are constructed in a specific decision context and users may not fully know their preferences beforehand. Within the scope of a decision process, preferences are strongly influenced by the goals of the customer, existing cognitive constraints, and the personal experience of the customer. Due to the fact that users do not have stable preferences, the interaction mechanisms provided by a recommender system and the information shown to a user can have an enormous impact on the outcome of a decision process.
Theories from decision psychology and cognitive psychology have already elaborated a number of methodological tools for explaining and predicting the user behavior in these scenarios. The major goal of this workshop is to establish a platform for industry and academia to present and discuss new ideas and research results that are related to the topic of human decision making in recommender systems. The workshop consists of a mix of six presentations of papers in which results of ongoing research as reported in these proceedings are presented and two invited talks: Bart Knijnenburg presenting “ Simplifying privacy decisions: towards interactive and adaptive solutions” and and Jill Freyne and Shlomo Berkovsky presenting: “Food Recommendations: Biases that Underpin Ratings”.
Many people today live in information-rich worlds, constantly facing the question: what should I do next? Which papers should I read to learn about a new area I am interested in? Which movie should I go to? Which restaurant would I like? The experience of friends and colleagues is a valuable resource for making such decisions, especially friends who are familiar with the subject area and have similar tastes.The field of recommender systems (or collaborative filtering) attempts to automate this process, e.g., by supporting people in making recommendations, finding a set of people who are likely to provide good recommendations for a given person, or deriving recommendations from implicit behavior such as browsing activity, buying patterns, and time on task.
Since their introduction in the early 1990's, automated recommender systems have revolutionized the marketing and delivery of commerce and content by providing personalized recommendations and predictions over a variety of large and complex product offerings. In this article, we review the key advances in col-laborative filtering recommender systems, focusing on the evolution from research concentrated purely on algorithms to research concentrated on the rich set of ques-tions around the user experience with the recommender. We show through examples that the embedding of the algorithm in the user experience dramatically affects the value to the user of the recommender. We argue that evaluating the user experience of a recommender requires a broader set of measures than have been commonly used, and suggest additional measures that have proven effective. Based on our analysis of the state of the field, we identify the most important open research problems, and outline key challenges slowing the advance of the state of the art, and in some cases limiting the relevance of research to real-world applications.
Various kinds of recommendation services open to the general public have recently been integrated into the website of the University Library of Karlsruhe as a test bed for information providers and e-commerce alike. This contribution reports on the development of RecoDiver, a graph-based user interface for behavior-based recommender systems. A Java applet integrated into the library's online catalog dynamically displays recommended further documents in a clickable graph centered around the document of interest to the user. A local view of the complete graph of recommendations is presented in a radial tree layout based on a minimum spanning tree with animated graph transitions featuring interpolations by polar coordinates to avoid crisscrossings. Further graph search tools like a selectable histogram of years of publication are available as well. This article portrays the user interface as well as the distributed web service architecture behind it and features an evaluation by user surveys showing the preference of users compared to the common lists of recommended items.
The amount of readily available on-line text has reached hundreds of billions of words and continues to grow. Yet for most core natural language tasks, algorithms continue to be optimized, tested and compared after training on corpora consisting of only one million words or less. In this paper, we evaluate the performance of different learning methods on a prototypical natural language disambiguation task, confusion set disambiguation, when trained on orders of magnitude more labeled data than has previously been used. We are fortunate that for this particular application, correctly labeled training data is free. Since this will often not be the case, we examine methods for effectively exploiting very large corpora when labeled data comes at a cost.
Twixonomy Visualization Interface: How to Wander Around User Preferences
- Di Tommaso
- G Stilo