Joseph A. Konstan

Joseph A. Konstan
University of Minnesota Twin Cities | UMN · Department of Computer Science and Engineering

About

135
Publications
71,022
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
29,265
Citations

Publications

Publications (135)
Article
Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly o...
Preprint
Full-text available
This paper proposes a vision and research agenda for the next generation of news recommender systems (RS), called the table d'hote approach. A table d'hote (translates as host's table) meal is a sequence of courses that create a balanced and enjoyable dining experience for a guest. Likewise, we believe news RS should strive to create a similar expe...
Article
Full-text available
Computational advertising (CA) is a rapidly growing field, but there are numerous challenges related to measuring its effectiveness. Some of these are classic challenges where CA offers a new aspect to the challenge (e.g., multi-touch attribution, bias), and some are brand-new challenges created by CA (e.g., fake data and ad fraud, creeping out cus...
Preprint
Full-text available
Computational advertising (CA) is a rapidly growing field, but there are numerous challenges related to measuring its effectiveness. Some of these are classic challenges where CA offers a new aspect to the challenge (e.g., multi touch attribution, bias), and some are brand new challenges created by CA (e.g., fake data and ad fraud, creeping out cus...
Preprint
Bulk email is a primary communication channel within organizations, with all-company messages and regular newsletters serving as a mechanism for making employees aware of policies, events, and other needed messages. Ineffective communication could result in substantial wasted employee time and lack of awareness or compliance. Previous studies on or...
Conference Paper
User-system interaction in recommender systems involves three aspects: temporal browsing (viewing recommendation lists and/or searching/filtering), action (performing actions on recommended items, e.g., clicking, consuming) and inaction (neglecting or skipping recommended items). Modern recommenders build machine learning models from recordings of...
Preprint
Full-text available
As the field of recommender systems has developed, authors have used a myriad of notations for describing the mathematical workings of recommendation algorithms. These notations ap-pear in research papers, books, lecture notes, blog posts, and software documentation. The dis-ciplinary diversity of the field has not contributed to consistency in not...
Article
Full-text available
In this study, we show that individual users’ preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual preferences for recommendation diversity, popularity and serendipity by measuring their personality...
Article
Full-text available
This paper reports on a study of 1840 users of the MovieLens recommender system with identified Big-5 personality types. Based on prior literature that suggests that personality type is a stable predictor of user preferences and behavior, we examine factors of user retention and engagement, content preferences, and rating patterns to identify recom...
Article
Full-text available
We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making...
Conference Paper
Temporally, users browse and interact with items in recommender systems. However, for most systems, the majority of the displayed items do not elicit any action from users. In other words, the user-system interaction process includes three aspects: browsing, action, and inaction. Prior recommender systems literature has focused more on actions than...
Article
Full-text available
This paper reports the findings of the Dagstuhl Perspectives Workshop 17442 on performance modeling and prediction in the domains of Information Retrieval, Natural language Processing and Recommender Systems. We present a framework for further research, which identifies five major problem areas: understanding measures, performance analysis, making...
Chapter
Recommender systems help users find information by recommending content that a user might not know about, but will hopefully like. Rating-based collaborative filtering recommender systems do this by finding patterns that are consistent across the ratings of other users. These patterns can be used on their own, or in conjunction with other forms of...
Conference Paper
Organizers of online groups often struggle to recruit members who can most effectively carry out the group's activities and remain part of the group over time. In a study of a sample of 30,000 new editors belonging to 1,054 English WikiProjects, we empirically examine the effects of generalized prior work-productivity experience (measured by overal...
Conference Paper
Full-text available
Over the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-addressed question. According to the most common definition, serendipity consists of three components:...
Conference Paper
This ¹ paper introduces and evaluates MovieExplorer, an interactive exploration tool designed to use the data available in a traditional ratings-based recommender system to provide an interactive interface more suited to user exploration and fulfillment of short-term recommendation needs. A field deployment with 1,950 users showed that users found...
Conference Paper
Recommender systems algorithms are generally evaluated primarily on machine learning criteria such as recommendation accuracy or top-n precision. In this work, we evaluate six recommendation algorithms from a user-centric perspective, collecting both objective user activity data and subjective user perceptions. In a field experiment involving 1508...
Article
Exploiting evidence that sporting results affect fans' mood, we analyze whether National Football League game outcomes can affect the contributions of Wikipedia editors who identify as fans of a specific team. We find that the day after a team loses, their fans decrease their contributions towards football-related pages (relative to after a win). R...
Article
As the gig economy continues to grow and freelance work moves online, five-star reputation systems are becoming more and more common. At the same time, there are increasing accounts of race and gender bias in evaluations of gig workers, with negative impacts for those workers. We report on a series of four Mechanical Turk-based studies in which par...
Conference Paper
The technical barriers for conversing with recommender systems using natural language are vanishing. Already, there are commercial systems that facilitate interactions with an AI agent. For instance, it is possible to say "what should I watch" to an Apple TV remote to get recommendations. In this research, we investigate how users initially interac...
Conference Paper
Current recommender systems often show the same most-highly recommended items again and again ignoring the feedback that users neither rate nor click on those items. We conduct an online field experiment to test two ways of manipulating top-N recommendations with the goal of improving user experience: cycling the top-N recommendation based on their...
Conference Paper
Prior work relevant to incorporating personality into recommender systems falls into two categories: social science studies and algorithmic ones. Social science studies of preference have found only small relationships between personality and category preferences, whereas, algorithmic approaches found a little improvement when incorporating persona...
Conference Paper
As users browse a recommender system, they systematically consider or skip over much of the displayed content. It seems obvious that these eye gaze patterns contain a rich signal concerning these users' preferences. However, because eye tracking data is not available to most recommender systems, these signals are not widely incorporated into person...
Conference Paper
We introduce a theoretical framework called precision crowdsourcing whose goal is to help turn online information consumers into information contributors. The framework looks at the timing and nature of the requests made of users and the feedback provided to users with the goal of increasing long-term contribution and engagement in the site or syst...
Conference Paper
Online communities suffer serious newcomer attrition. This paper explores whether and how early activity diversity – the degree to which a newcomer engages in a wide range of a site’s activities in the first session – is associated with their longevity. We introduce a metric (DSCORE) to characterize early activity diversity in online sites an...
Article
The MovieLens datasets are widely used in education, research, and industry. They are downloaded hundreds of thousands of times each year, reflecting their use in popular press programming books, traditional and online courses, and software. These datasets are a product of member activity in the MovieLens movie recommendation system, an active rese...
Conference Paper
Recommender systems are not one-size-fits-all; different algorithms and data sources have different strengths, making them a better or worse fit for different users and use cases. As one way of taking advantage of the relative merits of different algorithms, we gave users the ability to change the algorithm providing their movie recommendations and...
Conference Paper
Full-text available
Studies have shown that the recommendation of unseen, novel or serendipitous items is crucial for a satisfying and engaging user experience. As a result, recent developments in recommendation research have increasingly focused towards introducing novelty in user recommendation lists. While, existing solutions aim to find the right balance between t...
Article
In Fall 2013 we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive eva...
Conference Paper
Recent developments in user evaluation of recommender systems have brought forth powerful new tools for understanding what makes recommendations effective and useful. We apply these methods to understand how users evaluate recommendation lists for the purpose of selecting an algorithm for finding movies. This paper reports on an experiment in which...
Conference Paper
This panel explores the many roles of data analytics in today's cross-domain collaborations. In some instances, cross-domain analytics are required to understand big data. In others, big data holds the key to understanding and evaluating how people collaborate across domains. Panelists will present their experiences with big data and collaboration,...
Conference Paper
Eli Pariser coined the term 'filter bubble' to describe the potential for online personalization to effectively isolate people from a diversity of viewpoints or content. Online recommender systems - built on algorithms that attempt to predict which items users will most enjoy consuming - are one family of technologies that potentially suffers from...
Chapter
Research on online communities raises a number of challenges. It is difficult to get access to usage data, to users (to interview), and to the system itself to introduce new features (e.g., participation incentive mechanisms). One solution is for researchers to create an online community themselves. Although this provides more control and access, i...
Conference Paper
In Fall 2013 we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive eva...
Conference Paper
One of the goals of data-intensive research, in any field of study, is to grow knowledge over time as additional studies contribute to collective knowledge and understanding. Two steps are critical to making such research cumulative -- the individual research results need to be documented thoroughly and conducted on data made available to others (t...
Conference Paper
Full-text available
Although many off-line organizations give their employees training, mentorship, a cohort and other socialization experiences that improve their retention and productivity, online production communities rarely do this. This paper describes the planning, execution and evaluation of a socialization regime for an online technical support community. In...
Article
Community Question Answering (CQA) services enable their users to exchange knowledge in the form of questions and answers. These communities thrive as a result of a small number of highly active users, typically called experts, who provide a large number of high-quality useful answers. Expert identification techniques enable community managers to t...
Conference Paper
In this paper, we introduce the concept of question temporality as a measure of the usefulness of the answers provided on the questions asked in the Question Answering sites (QA). We define question temporality based on when the answers provided on the questions would expire. We use classification methods to show that the question temporality can b...
Article
Trust plays a critical role in our everyday lives from interpersonal communication to buyer-seller exchanges. Yet the concept of trust within the unique context of computer-mediated exchange (CME) has received little attention. To offer a theoretical contribution, we define key terms and synthesize relevant literature identifying four sources and s...
Conference Paper
Recommender systems research is being slowed by the difficulty of replicating and comparing research results. Published research uses various experimental methodologies and metrics that are difficult to compare. It also often fails to sufficiently document the details of proposed algorithms or the evaluations employed. Researchers waste time reimpl...
Conference Paper
Full-text available
Question answering communities (QA) are sustained by a handful of experts who provide a large number of high quality answers. Identifying these experts during the first few weeks of their joining the community can be beneficial as it would allow community managers to take steps to develop and retain these potential experts. In this paper, we explor...
Conference Paper
The ability to embed links to other resources in user generated content can help authors create more useful and usable content. A variety of interfaces have emerged for entity-linking at popular online sites; such interfaces vary in the way that entity linking is initiated (in-band or out-of-band with respect to the message creation), the timing of...
Article
Recommender systems are an important part of the information and e-commerce ecosystem. They represent a powerful method for enabling users to filter through large information and product spaces. Nearly two decades of research on collaborative filtering have led to a varied set of algorithms and a rich collection of tools for evaluating their perfor...
Conference Paper
Full-text available
Social question and answer (Q&A) sites receive tens of thousands of questions each day, producing a mix of archival information, asker satisfaction, and, sometimes, frustration. This paper builds upon several recent research efforts that have explored the nature and qualities of questions asked on these social Q&A sites by offering a focused examin...
Article
Full-text available
Social question and answer (Q&A) Web sites field a remarkable variety of questions: while one user seeks highly technical information, another looks to start a social exchange. Prior work in the field has adopted informal taxonomies of question types as a mechanism for interpreting user behavior and community outcomes. In this work, we contribute a...
Conference Paper
Community Question Answering (CQA) services enables users to ask and answer questions. In these communities, there are typically a small number of experts amongst the large population of users. We study which questions a user select for answering and show that experts prefer answering questions where they have a higher chance of making a valuable c...
Conference Paper
Tens of thousands of questions are asked and answered every day on social question and answer (Q&A) Web sites such as Yahoo Answers. While these sites generate an enormous volume of searchable data, the problem of determining which questions and answers are archival quality has grown. One major component of this problem is the prevalence of convers...
Conference Paper
Full-text available
Question and answer (Q&A) sites such as Yahoo! Answers are places where users ask questions and others answer them. In this paper, we investigate predictors of answer quality through a comparative, controlled field study of responses provided across several online Q&A sites. Along with several quantitative results concerning the effects of factors...
Chapter
We present PolyLens, a new collaborative filtering recommender system designed to recommend items for groups of users, rather than for individuals. A group recommender is more appropriate and useful for domains in which several people participate in a single activity, as is often the case with movies and restaurants. We present an analysis of the p...
Conference Paper
Full-text available
It is increasingly common for online communities to rely on members rather than editors to contribute and moderate content. To motivate members to perform these tasks, some sites display social comparisons, information designed to show members how they compare to others in the system. For example, Amazon, an online book store, shows a list of top r...
Conference Paper
Full-text available
We explore the use of social comparison theory as a natural mechanism to increase contributions to an online movie recommendation community by investigating the effects of social information on user behavior in an online field experiment. We find that, after receiving behavioral information about the median user's total number of movie ratings, use...
Conference Paper
Full-text available
Watching video online is becoming increasingly popular, and new video streaming technologies have the potential to transform video watching from a passive, isolating experience into an active, socially engaging experience. However, the viability of an active social experience is unclear: both chatting and watching video require attention, and may i...
Conference Paper
Full-text available
Spyware is an increasing problem. Interestingly, many programs carrying spyware honestly disclose the activities of the software, but users install the software anyway. We report on a study of software installation to assess the effectiveness of different notices for helping people make better decisions on which software to install. Our study of 22...
Article
This paper reports results from a controlled experiment (N = 50) measuring effects of interruption on task completion time, error rate, annoyance, and anxiety. The experiment used a sample of primary and peripheral tasks representative of those often performed by users. Our experiment differs from prior interruption experiments because it measures...
Article
Spyware is software which monitors user actions, gathers personal data, and/or displays advertisements to users. While some spyware is installed surreptitiously, a surprising amount is installed on users’ computers with their active participation. In some cases, users agree to accept spyware as part of a software bundle as a cost associated with ga...
Conference Paper
Full-text available
Recommender systems have shown great potential to help users find interesting and relevant items from within a large information space. Most research up to this point has focused on improving the accuracy of recommender systems. We believe that not only has this narrow focus been misguided, but has even been detrimental to the field. The recommenda...
Conference Paper
Full-text available
Recommender systems do no t always generate good recommendations for users. In order to improve recommender quality, we argue that recommenders need a deeper understanding of users and their information seeking tasks. Human-Recommender Interaction (HRI) provides a framework and a methodology for understanding users, their tasks, and recommender alg...
Conference Paper
Full-text available
If recommenders are to help people be more productive, they need to support a wide variety of real-world information seeking tasks, such as those found when seeking research papers in a digital library. There are many potential pitfalls, including not knowing what tasks to support, generating recommendations for the wrong task, or even failing to g...
Conference Paper
Full-text available
Automated recommender systems predict user preferences by applying machine learning techniques to data on products, users, and past user preferences for products. Such systems have become increasingly popular in entertainment and e-commerce domains, but have thus far had little success in information-seeking domains such as identifying published re...
Conference Paper
Full-text available
Economic modeling provides a formal mechanism to under- stand user incentives and behavior in online systems. In this paper we describe the process of building a parameterized economic model of user contributed ratings in an online movie recommender system. We con- structed a theoretical model to formalize our initial understanding of the system, a...
Conference Paper
Online communities (OLCs) are gatherings of like-minded people, brought together in cyberspace by shared interests. Creating such communities is not a big challenge; sustaining members' participation is. In this paper, we describe a technique for presenting members' photos and evaluate how it affects member participation in the community. We compar...
Conference Paper
This workshop intends to bring recommender systems researchers and practitioners together in order to discuss the current state of recommender systems research, both on existing and emerging research topics, and to determine how research in this area should proceed. We are at a pivotal point in recommender systems research where researchers are bot...
Article
Full-text available
Ratings-based recommender systems are one type of online community that relies on user contributions. We present an overview of the implicit incentive structures that motivate rating behavior in one such system, MovieLens. We conducted a survey of MovieLens users to determine their motivations, and formalized these findings in a parameterized econo...
Article
Recommender systems using collaborative filtering are a popular technique for reducing information overload and finding products to purchase. One limitation of current recommenders is that they are not portable. They can only run on large computers connected to the Internet. A second limitation is that they require the user to trust the owner of th...
Article
The World Wide Web has quickly become a primary source of information for a variety of real-time topics, such as news headlines, stock market data, sports scores, and weather forecasts. However, achieving a high degree of awareness for this real-time information is particularly challenging as users are often performing other necessary tasks. As a r...
Article
When an automating application needs a user's input or has feedback or other information for that user, it typically engages the user immediately, interrupting the user's current task. To empirically validate why unnecessarily interrupting a user's task should be avoided, we designed an experiment measuring the effects of an interruption on a user'...
Article
responsibility to the computer, coordinating the asynchronous interactions between the user and computer is becoming increasingly important. Without proper coordination, an application attempting to gain the user's attention risks interrupting the user in the midst of performing another task. To justify why an application should avoid interrupting...
Article
Full-text available
Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes ot...
Article
Full-text available
Recommender systems using collaborative filtering are a popular technique for reducing information overload and finding products to purchase. One limitation of current recommenders is that they are not portable. They can only run on large computers connected to the Internet. A second limitation is that they require the user to trust the owner of th...
Conference Paper
Full-text available
The number of research papers available is growing at a staggering rate. Researchers need tools to help them find the papers they should read among all the papers published each year. In this paper, we present and experiment with hybrid recommender algorithms that combine collaborative filtering and content-based filtering to recommend research pap...