Publications (5)11.83 Total impact
-
Article: Alternatives to peer review: novel approaches for research evaluation.
[show abstract] [hide abstract]
ABSTRACT: In this paper we review several novel approaches for research evaluation. We start with a brief overview of the peer review, its controversies, and metrics for assessing efficiency and overall quality of the peer review. We then discuss five approaches, including reputation-based ones, that come out of the research carried out by the LiquidPub project and research groups collaborated with LiquidPub. Those approaches are alternative or complementary to traditional peer review. We discuss pros and cons of the proposed approaches and conclude with a vision for the future of the research evaluation, arguing that no single system can suit all stakeholders in various communities.Frontiers in Computational Neuroscience 01/2011; 5:56. · 2.15 Impact Factor -
Conference Proceeding: UCount: a Community-Driven Approach for Measuring Scientific Reputation
Altmetrics Workshop. Web Science Conference 2011; 01/2011 -
Article: Solving the apparent diversity-accuracy dilemma of recommender systems.
[show abstract] [hide abstract]
ABSTRACT: Recommender systems use data on past user preferences to predict possible future likes and interests. A key challenge is that while the most useful individual recommendations are to be found among diverse niche objects, the most reliably accurate results are obtained by methods that recommend objects based on user or object similarity. In this paper we introduce a new algorithm specifically to address the challenge of diversity and show how it can be used to resolve this apparent dilemma when combined in an elegant hybrid with an accuracy-focused algorithm. By tuning the hybrid appropriately we are able to obtain, without relying on any semantic or context-specific information, simultaneous gains in both accuracy and diversity of recommendations.Proceedings of the National Academy of Sciences 02/2010; 107(10):4511-5. · 9.68 Impact Factor -
Article: The effect of discrete vs. continuous-valued ratings on reputation and ranking systems
[show abstract] [hide abstract]
ABSTRACT: When users rate objects, a sophisticated algorithm that takes into account ability or reputation may produce a fairer or more accurate aggregation of ratings than the straightforward arithmetic average. Recently a number of authors have proposed different co-determination algorithms where estimates of user and object reputation are refined iteratively together, permitting accurate measures of both to be derived directly from the rating data. However, simulations demonstrating these methods' efficacy assumed a continuum of rating values, consistent with typical physical modelling practice, whereas in most actual rating systems only a limited range of discrete values (such as a 5-star system) is employed. We perform a comparative test of several co-determination algorithms with different scales of discrete ratings and show that this seemingly minor modification in fact has a significant impact on algorithms' performance. Paradoxically, where rating resolution is low, increased noise in users' ratings may even improve the overall performance of the system. Comment: 6 pages, 2 figures01/2010; -
Article: State of the Art in Scientific Knowledge Creation, Dissemination, Evaluation and Maintenance
[show abstract] [hide abstract]
ABSTRACT: This report presents an overview of the State of the Art in topics related to the on-going research in the Liquid Publications Project. The Liquid Publications Project (LiquidPub) aims to bring fundamental changes to the processes by which scientific knowledge is created, disseminated, evaluated and maintained. In order to accomplish this, many processes and areas will have to be modified. We group the areas involved in this change into four areas: creation and evolution of scientific knowledge, evaluation processes (primarily, peer-review processes and their evaluation), computational trust and reputation mechanisms, and business and process models. Due to the size and complexity of each of these four areas, we will only discuss topics that are directly related to our proposed research.
