No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Case study: Recommending course reading materials in a small virtual learning community
Abstract
Recommender systems, long used in e-commerce to help users find salient items, also offer tools for virtual learning communities to let the community determine what items are most pertinent to its members. However, due to differences in numbers and goals, learning environments cannot simply copy e-commerce approaches to recommenders. This article discusses design issues related to using recommenders in learning environments and student perceptions of using rating and commenting to allow students to winnow additional reading materials in a university course website. Positive student perceptions show that recommenders can enhance virtual learning community experience. The rating feature in particular was viewed positively and perceived to influence selecting behaviour, while commenting, although also perceived positively, was seen as underused and less influential. In addition, the design of the system is evaluated in the light of the student feedback and potential improvements are discussed.
... The requirements are: a recommendation model; an open standards-based service-oriented architecture; and a usable and accessible graphical user interface to deliver the recommendations. Leino [32] discussed design issues associated to employing recommenders in learning environments and how student perceptions of using rating and commenting can affect students in winnowing additional reading materials in a university course website. Positive student perceptions show that recommenders can improve the experience in virtual learning community. ...
... improve virtual community experiences [32]. ...
With the rapid increasing of learning objects (LOs) in a variety of media formats, it becomes quite difficult and complicated task for learners to find suitable LOs based on their needs and preferences. To support personalization, recommender systems can be used to assist learners in finding the appropriate LOs which will be needed for their learning. In this paper, we propose a framework of a semantic recommender system for e-learning in which it will assist learners to find and select the relevant LOs to their field of interest. The proposed framework utilizes the intra and extra semantic relationships between LOs and the learner's needs to provide personalized recommendations for learners. The semantic recommendation algorithm is based on the extension of the query keywords by using the semantic relations, concepts and reasoning means in the domain ontology. The proposed system can be used to reduce the time and effort involved in finding suitable LOs, and thus, improves the quality of learning.
... Additionally, in [RS69-2013] learning effectiveness, learning efficiency, course engagement and knowledge acquisition were measured to evaluate recommendations impact in a MOOC [89]. The study on learners perception as reported in [RS61-2012] suggests that recommenders can significantly enhance virtual learning communities and put the power of determining what constitutes a quality contribution in the hands of the community members [56]. ...
This chapter presents an analysis of recommender systems in Technology-Enhanced Learning along their 15 years existence (2000–2014). All recommender systems considered for the review aim to support educational stakeholders by personalising the learning process. In this meta-review 82 recommender systems from 35 different countries have been investigated and categorised according to a given classification framework. The reviewed systems have been classified into seven clusters according to their characteristics and analysed for their contribution to the evolution of the RecSysTEL research field. Current challenges have been identified to lead the work of the forthcoming years.
... Additionally , in learning effectiveness, learning efficiency, course engagement and knowledge acquisition were measured to evaluate recommendations impact in a MOOC [88]. The study on learners perception as reported in [RS61-2012] suggests that recommenders can significantly enhance virtual learning communities and put the power of determining what constitutes a quality contribution in the hands of the community members [55]. [RS26-2009] evaluated the applicability of recommendations in mash-up environments that combine sources of users from different Web2.0 services [22]. ...
This chapter presents an analysis of recommender systems in Technology-Enhanced Learning along their 15 years existence (2000–2014). All recommender systems considered for the review aim to support educational stakeholders by personalising the learning process. In this meta-review 82 recommender systems from 35 different countries have been investigated and categorised according to a given classification framework. The reviewed systems have been classified into seven clusters according to their characteristics and analysed for their contribution to the evolution of the RecSysTEL research field. Current challenges have been identified to lead the work of the forthcoming years.
... propagates the "word-of-mouth" from trusted and high quality resources (Recker & Walker, 2003), and enhances virtual community experiences (Leino, 2012). ...
With the growing number of students in the classroom and the switch to online environments, instructors are beginning to integrate collaborative learning approaches in the classroom. However, many times in large collaborative environments and large social networks, students are overwhelmed by the amount of available information; it is often challenging to select the most appropriate sources of information. A promising way to deal with this challenge and enhance social interaction in collaborative learning environments is by introducing recommender systems. The main goal of this article is, through a literature review, explore the differences between general recommender systems and educational recommender systems, and to provide a general overview about the benefits, challenges and limitations of recommender systems in educational settings.
... Our discussion is based on student e-questionnaire replies and actual click-by-click use data. The 2009 experiences of binary ratings and commenting have already been discussed in Leino (2011) and Leino (2012). Consequently, here we focus on experiences of five-star ratings (2011) vis-à-vis binary ratings (2009)(2010). ...
As various recommender approaches are increasingly considered in e-learning, the need for actual use cases to guide development efforts is growing. We report on our experiences of using non-algorithmic recommender features to recommend additional study materials on an undergraduate course in 2009--2011. The study data comes from student e-questionnaire replies and actual click-by-click use data. Our discussion centres on using binary (useful/not useful) rating scale (2009--2010) vis-à-vis five-star rating scale (2011). Using five-star scale to increase the complexity of the rating decision significantly reduced dishonesty (rating items without viewing them), but at the price of fewer ratings overall and increased complexity of interpreting the ratings. In addition to explaining how ratings and other factors inter-influenced item-selecting, we also discuss how different scales (binary and five-star) affect the rating behaviour in e-learning and how the five-star rating distributions in e-learning relate to those in other domains. Furthermore, we discuss two models, high-quality approach and low-cost approach, of employing non-algorithmic recommending features in e-learning that emerge from our findings. The findings provide the field with insight into the actual dynamics of using recommender features in e-learning. Moreover, they provide practitioners with actionable information on dishonesty.
Offline evaluations using historical data offer a fast and repeatable way to evaluate TEL recommender systems. However, this is only possible if historical datasets contain all particular information needed by the recommendation algorithm. Another challenge is users must have indicated interest in the recommended resource in the historical data collected for a resource to be evaluated as relevant. This however does not mean the user would not be interested in this newly recommended resource. User experiments help to complement offline evaluations but due to the effort and costs of performing these experiments, very few are conducted. Crowdsourcing is a solution to this challenge as it gives access to sufficient willing users. This paper investigates the evaluation of a graph-based recommender system for TEL using crowdsourcing. Initial results show that crowdsourcing can indeed be used as an evaluation method for TEL recommender systems.
The increasing number of publications on recommender systems for Technology Enhanced Learning (TEL) evidence a growing interest in their development and deployment. In order to support learning, recommender systems for TEL need to consider specific requirements, which differ from the requirements for recommender systems in other domains like e-commerce. Consequently, these particular requirements motivate the incorporation of specific goals and methods in the evaluation process for TEL recommender systems. In this article, the diverse evaluation methods that have been applied to evaluate TEL recommender systems are investigated. A total of 235 articles are selected from major conferences, workshops, journals and books where relevant work have been published between 2000 and 2014. These articles are quantitatively analysed and classified according to the following criteria: type of evaluation methodology, subject of evaluation, and effects measured by the evaluation. Results from the survey suggest that there is a growing awareness in the research community of the necessity for more elaborate evaluations. At the same time, there is still substantial potential for further improvements. This survey highlights trends and discusses strengths and shortcomings of the evaluation of TEL recommender systems thus far, thereby aiming to stimulate researchers to contemplate novel evaluation approaches.
Social online learning environments provide new recommendation opportunities to meet users' needs. However, current educational recommender systems do not usually take advantage of these opportunities. To progress on this issue, we have proposed a knowledge engineering approach based on human–computer interaction (i.e. user-centred design as defined by the standard ISO 9241-210:2010) and artificial intelligence techniques (i.e. data mining) that involve educators in the process of eliciting educational oriented recommendations. To date, this approach differs from most recommenders in education in focusing on identifying relevant actions to be recommended on e-learning services from a user-centric perspective, thus widening the range of recommendation types. This approach has been used to identify 32 recommendations that consider several types of actions, which focus on promoting active participation of learners and on strengthening the sharing of experiences among peers through the usage of the social services provided by the learning environment. The paper describes where data mining techniques have been applied to complement the user-centred design methods to produce social oriented recommendations in online learning environments.
In this paper we address an open key issue during the development of web-based educational systems. In particular, we provide an educational-oriented approach for building personalised e-learning environments that focuses on putting the learnersʼ needs in the centre of the development process. Our approach proposes user centred design methodologies involving interdisciplinary teams of software developers and domain experts. It is illustrated in an adaptive e-learning system, where a MOOC (Massive Open Online Course) was taken by nearly 400 learners. In particular, we report where user centred design methods can be applied along the e-learning life cycle to designing and evaluating personalisation support through recommendations in learning management systems.
We study the problem of designing a mechanism to rank items in forums by making use of the user reviews such as thumb and star ratings. We compare mechanisms where fo- rum users rate individual posts and also mechanisms where the user is asked to perform a pairwise comparison and state which one is better. The main metric used to evaluate a mechanism is the ranking accuracy vs the cost of reviews, where the cost is measured as the average number of reviews used per post. We show that for many reasonable prob- ability models, there is no thumb (or star) based ranking mechanism that can produce approximately accurate rank- ings with bounded number of reviews per item. On the other hand we provide a review mechanism based on pair- wise comparisons which achieves approximate rankings with bounded cost. We have implemented a system, shoutveloc- ity (5), which is a twitter-like forum but items (i.e., tweets in Twitter) are rated by using comparisons. For each new item the user who posts the item is required to compare two previous entries. This ensures that over a sequence of n posts, we get at least n comparisons requiring one review per item on average. Our mechanism uses this sequence of comparisons to obtain a ranking estimate. It ensures that every item is reviewed at least once and winning entries are reviewed more often to obtain better estimates of top items.
We examine tagging behavior on Flickr, a public photo- sharing website. We build on previous qualitative research that exposed a taxonomy of tagging motivations, as well as on social presence research. The taxonomy suggests that motivations for tagging are tied to the intended target audience of the tags - the users themselves, family and friends, or the general public. Using multiple data sources, including a survey and independent system data, we examine which motivations are associated with tagging level, and estimate the magnitude of their contribution. We find that the levels of the Self and Public motivations, together with social presence indicators, are positively correlated with tagging level; Family & Friends motivations are not significantly correlated with tagging. The findings and the use of survey method carry implications for designers of tagging and other social systems on the web.
We studied user behavior in a recommender-rich environment, Amazon online store, to see what role the algorithm-based and user-generated recommendations play in finding items of interest. We used applied ethnography, on-location interviewing and observation, to get an accurate picture of user activity. We were especially interested in the role of customer ratings and reviews and what kind of strategies users had developed for such an environment. Our results underline the need to develop recommender systems as a whole. The way the recommendations are shown affects which items get picked, and for improving the interface, it is necessary to study the whole in addition to studying the parts in isolation.
Tagging has emerged as a powerful mechanism that enables users to find, organize, and understand online entities. Recommender systems similarly enable users to efficiently navigate vast collec- tions of items. Algorithms combining tags with recommenders may deliver both the automation inherent in recommenders, and the flexibility and conceptual comprehensibility inherent in tagging systems. In this paper we explore tagommenders, recommender al- gorithms that predict users' preferences for items based on their inferred preferences for tags. We describe tag preference inference algorithms based on users' interactions with tags and movies, and evaluate these algorithms based on tag preference ratings collected from 995 MovieLens users. We design and evaluate algorithms that predict users' ratings for movies based on their inferred tag prefer- ences. Our tag-based algorithms generate better recommendation rankings than state-of-the-art algorithms, and they may lead to flex- ible recommender systems that leverage the characteristics of items users find most important.
Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.
Recommender systems are changing from novelties used by a few E-commerce sites, to serious business tools that are re-shaping the world of E-commerce. Many of the largest commerce Web sites are already using recommender systems to help their customers find products to purchase. A recommender system learns from a customer and recommends products that she will find most valuable from among the available products. In this paper we present an explanation of how recommender systems help E-commerce sites increase sales, and analyze six sites that use recommender systems including several sites that use more than one recommender system. Based on the examples, we create a taxonomy of recommender systems, including the interfaces they present to customers, the technologies used to create the recommendations, and the inputs they need from customers. We conclude with ideas for new applications of recommender systems to E-commerce. Keywords Electronic commerce, recommender systems,...
The increasing amount of content available via digital television has made TV program recommenders valuable tools. In order to provide personalized recommendations, recommender systems need to collect information about user preferences. Since users are reluctant to invest much time in explicitly expressing their interests, preferences often need to be implicitly inferred through data gathered by monitoring user behavior. Which is, alas, less reliable. This article addresses the problem of learning TV preferences based on tracking the programs users have watched, whilst dealing with the varying degrees of reliability in such information. Three approaches to the problem are discussed: use all information equally; weight information by its reliability or simply discard the most unreliable information. Experimental results for these three approaches are presented and compared using a content-based filtering recommender built on a Naïve Bayes classifier.
With recommender systems, users receive items recommended on the basis of their profile. New users experience the cold start problem: as their profile is very poor, the system performs very poorly. In this paper, classical new user cold start techniques are improved by exploiting the cold user data, i.e. the user data that is readily available (e.g. age, occupation, location, etc.), in order to automatically associate the new user with a better first profile. Relying on the existing �-community spaces model, a rule-based induction process is used and a recommendation process based on the "level of agreement" principle is defined. The experiments show that the quality of recommendations compares to that obtained after a classical new user technique, while the new user effort is smaller as no initial ratings are asked.
Collaborative tagging provides exceptional performance in the domains of IF (Information Filtering) and IR (Information Retrieval). Based on various studies regarding the tagging behavior of users, it can be concluded that there is potential for expansion of this domain to the area of ratings. The paper presents Qtag, a qualitative tagging system that allows users to tag in order to rate and express opinions using a sharable vocabulary. A conceptual model and evaluation are presented. Tag visualization model which is under-construction is also presented.
Ten rules for web startups', Evan Williams' blog, available at http://evhead.com
- E Williams
News Narratives and News Framing: Constructing Political Reality, Rowman &
- K S Johnson-Cartee