Conference Paper

# CyberRank: Knowledge Elicitation for Risk Assessment of Database Security

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

## Abstract

Security systems for databases produce numerous alerts about anomalous activities and policy rule violations. Prioritizing these alerts will help security personnel focus their efforts on the most urgent alerts. Currently, this is done manually by security experts that rank the alerts or define static risk scoring rules. Existing solutions are expensive, consume valuable expert time, and do not dynamically adapt to changes in policy. Adopting a learning approach for ranking alerts is complex due to the efforts required by security experts to initially train such a model. The more features used, the more accurate the model is likely to be, but this will require the collection of a greater amount of user feedback and prolong the calibration process. In this paper, we propose CyberRank, a novel algorithm for automatic preference elicitation that is effective for situations with limited experts' time and outperforms other algorithms for initial training of the system. We generate synthetic examples and annotate them using a model produced by Analytic Hierarchical Processing (AHP) to bootstrap a preference learning algorithm. We evaluate different approaches with a new dataset of expert ranked pairs of database transactions, in terms of their risk to the organization. We evaluated using manual risk assessments of transaction pairs, CyberRank outperforms all other methods for cold start scenario with error reduction of 20%.

## No full-text available

... To address this challenge, we suggest incorporating the concept of diversity from recommendation systems (Matt et al. 2014) into logging policies. Unlike search engines or recommendations, sampling a more diverse group of users is not technically complicated as the user's transactions risk can be aggregated to a single score (Grushka-Cohen et al. 2016;Evina et al. 2019). However, logging capacity is constrained, and by focusing solely on diversity, undocumented malicious activity in the high risk group can be missed. ...
... When an SO assigns risk to a transaction, various contextual information is used, such as time of day, user activity profile, location (IP address), the nature of the activity (i.e. is it permitted), data sensitivity, and the resulting data volume. When a DAM system is installed in an organization these rules can be defined manually (by the SO) as a risk policy or learned by annotating risk scores on some representative transaction using a classifier such as CyberRank (Grushka-Cohen et al. 2016). ...
... To reduce the complexity of the features for comparison and evaluation, working with aggregated data is useful. Previous work such as (Grushka-Cohen et al. 2016;Evina et al. 2019) leveraged SO knowledge to aggregate database activity into a single risk score. (Grushka-Cohen et al. 2019) suggested a simulation package made of low complexity data where the user activity for a time frame is represented by a single aggregated risk score. ...
Preprint
Full-text available
Database activity monitoring (DAM) systems are commonly used by organizations to protect the organizational data, knowledge and intellectual properties. In order to protect organizations database DAM systems have two main roles, monitoring (documenting activity) and alerting to anomalous activity. Due to high-velocity streams and operating costs, such systems are restricted to examining only a sample of the activity. Current solutions use policies, manually crafted by experts, to decide which transactions to monitor and log. This limits the diversity of the data collected. Bandit algorithms, which use reward functions as the basis for optimization while adding diversity to the recommended set, have gained increased attention in recommendation systems for improving diversity. In this work, we redefine the data sampling problem as a special case of the multi-armed bandit (MAB) problem and present a novel algorithm, which combines expert knowledge with random exploration. We analyze the effect of diversity on coverage and downstream event detection tasks using a simulated dataset. In doing so, we find that adding diversity to the sampling using the bandit-based approach works well for this task and maximizing population coverage without decreasing the quality in terms of issuing alerts about events.
... Furthermore, even if the transaction data is compressed, the sheer volume of transactions in corporate or government grade production systems prohibits saving all of the data, particularly due to cost considerations. Techniques for sampling and their effects on anomaly detection have been studies in the domain of network traffic flow [4,7,11] and the domain of Cyber security for Internet page retrieval. However, these domains are quite different from the domain of database transaction as the data is richer, containing more features, and the damage from a single transaction can be greater than the damage from a network packet. ...
... It has been established that sampling introduces bias to anomaly detection. In a previous work [4] we suggested a Gibbs sampling approach using the transaction risk as the prior for sampling it, an approach we test using the new simulation environment. ...
... We aimed to simulate a system where for each time frame users transactions are represented by the risk of his activity during that time frame. The risk can be assessed for transactions using a rule based scoring policy or a ranking approach such as CyberRank [4]. According to security experts interviewed during the development of the simulation users behavior is not random, but has trends both in the activity volume and the risk the activity presents. ...
Chapter
Monitoring database activity is useful for identifying and preventing data breaches. Such database activity monitoring (DAM) systems use anomaly detection algorithms to alert security officers to possible infractions. However, the sheer number of transactions makes it impossible to track each transaction. Instead, solutions use manually crafted policies to decide which transactions to monitor and log. Creating a smart data-driven policy for monitoring transactions requires moving beyond manual policies. In this paper, we describe a novel simulation method for user activity. We introduce events of change in the user transaction profile and assess the impact of sampling on the anomaly detection algorithm. We found that looking for anomalies in a fixed subset of the data using a static policy misses most of these events since low-risk users are ignored. A Bayesian sampling policy identified 67% of the anomalies while sampling only 10% of the data, compared to a baseline of using all of the data.
... We propose using a sampling strategy based on the perceived risk posed by each transaction to the organization. The risk can be estimated using a manually calibrated policy or estimated using a machine learning ranking algorithm such as CyberRank [12]. ...
... The users can be ether application data base user or a real user. As described in our CyberRank work [12], the user is an important entity whose behavior and activity is useful for identifying risk and controlling database transactions. When a transaction occurs, it is compared to the user's history to detect anomalies. ...
... The risk captures the likelihood that an SO would investigate the anomaly [12], however the investigation will be more thorough if low-risk transactions are also captured for the suspect user. ...
Article
Full-text available
Data leakage and theft from databases is a dangerous threat to organizations. Data Security and Data Privacy protection systems (DSDP) monitor data access and usage to identify leakage or suspicious activities that should be investigated. Because of the high velocity nature of database systems, such systems audit only a portion of the vast number of transactions that take place. Anomalies are investigated by a Security Officer (SO) in order to choose the proper response. In this paper we investigate the effect of sampling methods based on the risk the transaction poses and propose a new method for "combined sampling" for capturing a more varied sample.
... For this third survey phase, ACM Digital Library citation database was additionally included in order to make the research more comprehensive and accurate. Only 2 relevant papers were found in ACM Digital Library [36,37]. ...
... According to the results, the new risk model reduces the number of incidents and allows security analysts to focus solely on a smaller number of actual and critical incidents, which consequently reduces the time and resources. In an article [36], a new algorithm for ranking cyber security alerts for databases is proposed. The goal was to develop an AHP prioritization method that can automatically rank alerts at the level of risk posed by a particular transaction, thus allowing the security professionals to focus their time and efforts on the most important alerts. ...
... In this task, the model's goal is to predict the user who submitted the query containing a particular operator. While the DBMS generally knows the user submitting a query, such a classifier is useful for determining when a user-submitted query does not match the queries usually submitted by that user, a common learning task in database intrusion detection [6,18,23,33]. ...
... SageDB [57] proposes integrating machine learning techniques into join processing, sorting, and indexing. Recent works in intrusion detection [6,18], index structures [26], SLA management [15,32,35,36,44,45,56], entity matching [42], physical design [39,47], and latency prediction [2,13,14,16,30,[60][61][62] have all employed machine learning techniques. With little exception, each of these works have included hand-engineered features derived for each particular task. ...
Preprint
Integrating machine learning into the internals of database management systems requires significant feature engineering, a human effort-intensive process to determine the best way to represent the pieces of information that are relevant to a task. In addition to being labor intensive, the process of hand-engineering features must generally be repeated for each data management task, and may make assumptions about the underlying database that are not universally true. We introduce flexible operator embeddings, a deep learning technique for automatically transforming query operators into feature vectors that are useful for a multiple data management tasks and is custom-tailored to the underlying database. Our approach works by taking advantage of an operator's context, resulting in a neural network that quickly transforms sparse representations of query operators into dense, information-rich feature vectors. Experimentally, we show that our flexible operator embeddings perform well across a number of data management tasks, using both synthetic and real-world datasets.
... Risk score can be assigned based on predefined list of rules and conditions. The science community has been extensively working on using machine learning models for risk scoring in many domains [18,6,4] For an ML risk score to be accepted by the medical community it is important that it allows experts to tweak it (add clinical features, local biases) and is explainable [14]. Clalit Health Services (CHS) created a risk-scoring tool to predict the severity of COVID-19. ...
Preprint
Full-text available
Testing is an important part of tackling the COVID-19 pandemic. Availability of testing is a bottleneck due to constrained resources and effective prioritization of individuals is necessary. Here, we discuss the impact of different prioritization policies on COVID-19 patient discovery and the ability of governments and health organizations to use the results for effective decision making. We suggest a framework for testing that balances the maximal discovery of positive individuals with the need for population-based surveillance aimed at understanding disease spread and characteristics. This framework draws from similar approaches to prioritization in the domain of cyber-security based on ranking individuals using a risk score and then reserving a portion of the capacity for random sampling. This approach is an application of Multi-Armed-Bandits maximizing exploration/exploitation of the underlying distribution. We find that individuals can be ranked for effective testing using a few simple features, and that ranking them using such models we can capture 65% (CI: 64.7%-68.3%) of the positive individuals using less than 20% of the testing capacity or 92.1% (CI: 91.1%-93.2%) of positives individuals using 70% of the capacity, allowing reserving a significant portion of the tests for population studies. Our approach allows experts and decision-makers to tailor the resulting policies as needed allowing transparency into the ranking policy and the ability to understand the disease spread in the population and react quickly and in an informed manner.
... In other cases, the risk assessment is sometimes discussed. Others proposed to include risk assessment solutions in access control systems for verifying the risk level associated with access requests before according authorizations [5], [6]. We, among other authors are thoroughly studying the issue and discuss about correlation so that, once detected, these anomalies can be analyzed to determine the correlation between them. ...
Chapter
Database activity monitoring systems aim to protect organizational data by logging users’ activity to Identify and document malicious activity. High-velocity streams and operating costs, restrict these systems to examining only a sample of the activity. Current solutions use manual policies to decide which transactions to monitor. This limits the diversity of the data collected, creating a “filter bubble” over representing specific subsets of the data such as high-risk users and under-representing the rest of the population which may never be sampled. In recommendation systems, Bandit algorithms have recently been used to address this problem. We propose addressing the sampling for database activity monitoring problem as a recommender system. In this work, we redefine the data sampling problem as a special case of the multi-armed bandit problem and present a novel algorithm, C–$$\epsilon$$–Greedy, which combines expert knowledge with random exploration. We analyze the effect of diversity on coverage and downstream event detection using simulated data. In doing so, we find that adding diversity to the sampling using the bandit-based approach works well for this task, maximizing population coverage without decreasing the quality in terms of issuing alerts about events, and outperforming policies manually crafted by experts and other sampling methods.
Conference Paper
Full-text available
As a main method in database intrusion detection, database anomaly detection should be able to detect users' operational behaviours for timely prevention of possible attacks and for guarantee of database security. Aiming at this, we apply cluster analysis techniques to anomaly detection and propose a novel density-based clustering algorithm called DBCAPSIC, which is adopted to clustering database users according to their behavior types and behavior frequencies. Privilege patterns are extracted from the clusters and serve as a reference in anomaly detection. The simulation experiment proves that the algorithm can recognize the anomalous operations with few mistakes.
Conference Paper
Full-text available
Ranking SVM, which formalizes the problem of learning a ranking model as that of learning a binary SVM on preference pairs of documents, is a state-of-the-art ranking model in information retrieval. The dual form solution of Ranking SVM model can be written as a linear combination of the preference pairs, i.e., w = (i,j) αij(xi − xj), where αij denotes the Lagrange parameters associated with each pair (i, j). It is obvious that there exist significant interactions over the document pairs because two preference pairs could share a same document as their items. Thus it is natural to ask if there also exist interactions over the model parameters αij, which we may leverage to propose better ranking model. This paper aims to answer the question. Firstly, we found that there exists a low-rank structure over the Ranking SVM model parameters αij, which indicates that the interactions do exist. Then, based on the discovery, we made a modification on the original Ranking SVM model by explicitly applying a low-rank constraint to the parameters. Specifically, each parameter αij is decomposed as a product of two low-dimensional vectors, i.e., αij = vi, vj, where vectors vi and vj correspond to document i and j, respectively. The learning process, thus, becomes to optimize the modified dual form objective function with respect to the low-dimensional vectors. Experimental results on three LETOR datasets show that our method, referred to as Fac-torized Ranking SVM, can outperform state-of-the-art base-lines including the conventional Ranking SVM.
Conference Paper
Full-text available
The disclosure of sensitive data to unauthorized entities is a critical issue for organizations. Timely detection of data leakage is crucial to reduce possible damages. Therefore, breaches should be detected as early as possible, e.g., when data are leaving the database. In this paper, we focus on data leakage detection by monitoring database activities. We present a framework that automatically learns normal user behavior, in terms of database activities, and detects anomalies as deviation from such behavior. In addition, our approach explicitly indicates the root cause of an anomaly. Finally, the framework assesses the severity of data leakages based on the sensitivity of the disclosed data.
Article
Full-text available
Detecting and preventing data leakage and data misuse poses a serious challenge for organizations, especially when dealing with insiders with legitimate permissions to access the organization's systems and its critical data. In this paper, we present a new concept, Misuseability Weight, for estimating the risk emanating from data exposed to insiders. This concept focuses on assigning a score that represents the sensitivity level of the data exposed to the user and by that predicts the ability of the user to maliciously exploit this data. Then, we propose a new measure, the M-score, which assigns a misuseability weight to tabular data, discuss some of its properties, and demonstrate its usefulness in several leakage scenarios. One of the main challenges in applying the M-score measure is in acquiring the required knowledge from a domain expert. Therefore, we present and evaluate two approaches toward eliciting misuseability conceptions from the domain expert.
Chapter
Full-text available
Recommender Systems (RSs) are software tools and techniques providing suggestions for items to be of use to a user. In this introductory chapter we briefly discuss basic RS ideas and concepts. Our main goal is to delineate, in a coherent and structured way, the chapters included in this handbook and to help the reader navigate the extremely rich and detailed content that the handbook offers.
Conference Paper
Full-text available
We investigate using gradient descent meth- ods for learning ranking functions; we pro- pose a simple probabilistic cost function, and we introduce RankNet, an implementation of these ideas using a neural network to model the underlying ranking function. We present test results on toy data and on data from a commercial internet search engine.
Conference Paper
Full-text available
The paper is concerned with learning to rank, which is to construct a model or a function for ranking objects. Learning to rank is useful for document retrieval, collaborative filtering, and many other applications. Several methods for learning to rank have been proposed, which take object pairs as 'instances' in learning. We refer to them as the pairwise approach in this paper. Al- though the pairwise approach offers advantages, it ignores the fact that ranking is a prediction task on list of objects. The paper postulates that learn- ing to rank should adopt the listwise approach in which lists of objects are used as 'instances' in learning. The paper proposes a new proba- bilistic method for the approach. Specifically it introduces two probability models, respectively referred to as permutation probability and top k probability, to define a listwise loss function for learning. Neural Network and Gradient Descent are then employed as model and algorithm in the learning method. Experimental results on infor- mation retrieval show that the proposed listwise approach performs better than the pairwise ap- proach.
Conference Paper
Full-text available
This paper aims to conduct a study on the listwise approach to learning to rank. The listwise approach learns a ranking function by taking individual lists as instances and minimizing a loss function defined on the predicted list and the ground-truth list. Existing work on the approach mainly focused on the development of new algorithms; methods such as RankCosine and ListNet have been proposed and good performances by them have been observed. Unfortunately, the underlying theory was not sufficiently studied so far. To amend the problem, this paper proposes conducting theoretical analysis of learning to rank algorithms through investigations on the properties of the loss functions, including consistency, soundness, continuity, differentiability, convexity, and efficiency. A sufficient condition on consistency for ranking is given, which seems to be the first such result obtained in related research. The paper then conducts analysis on three loss functions: likelihood loss, cosine loss, and cross entropy loss. The latter two were used in RankCosine and ListNet. The use of the likelihood loss leads to the development of a new listwise method called ListMLE, whose loss function offers better properties, and also leads to better experimental results.
Article
Full-text available
Outlier detection has been used for centuries to detect and, where appropriate, remove anomalous observations from data. Outliers arise due to mechanical faults, changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. Their detection can identify system faults and fraud before they escalate with potentially catastrophic consequences. It can identify errors and remove their contaminating effect on the data set and as such to purify the data for processing. The original outlier detection methods were arbitrary but now, principled and systematic techniques are used, drawn from the full gamut of Computer Science and Statistics. In this paper, we introduce a survey of contemporary techniques for outlier detection. We identify their respective motivations and distinguish their advantages and disadvantages in a comparative review.
Article
Full-text available
to difierentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the efiectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the difierent existing techniques in that category are variants of the basic tech- nique. This template provides an easier and succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the difierent directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with.
Article
Attempts by insiders to exfiltrate data have become a severe threat to the enterprise. Conventional data security techniques, such as access control and encryption, must be augmented with techniques to detect anomalies in data access that may indicate exfiltration attempts. In this paper, we present the design and evaluation of DBSAFE, a system to detect, alert on, and respond to anomalies in database access designed specifically for relational database management systems (DBMS). The system automatically builds and maintains profiles of normal user and application behavior, based on their interaction with the monitored database during a training phase. The system then uses these profiles to detect anomalous behavior that deviates from normality. Once an anomaly is detected, the system uses predetermined policies guiding automated and/or human response to the anomaly. The DBSAFE architecture does not impose any restrictions on the type of the monitored DBMS. Evaluation results indicate that the proposed techniques are indeed effective in detecting anomalies.
Article
Transductive learning is a semi-supervised learning paradigm that can leverage unlabeled data by creating pseudo labels for learning a ranking model, when there is only limited or no training examples available. However, the effectiveness of transductive learning in information retrieval (IR) can be hindered by the low quality pseudo labels. To this end, we propose to incorporate a two-step k-means clustering algorithm to select the high quality training queries for generating the pseudo labels. In particular, the first step selects the high-quality queries for which the relevant documents are highly coherent as indicated by the clustering results. The second step then selects the initial training examples for the transductive learning that iteratively aggregating the pseudo examples. Finally, the learning to rank (LTR) algorithms are applied to learn the ranking model using the pseudo training examples created by the transductive learning process. Our proposed approach is particularly suitable for applications where there is only little or no human labels available as it does not necessarily involve the use of relevance assessments information or human efforts. Experimental results on the standard TREC Tweets11 collection show that our proposed approach outperforms strong baselines, namely the conventional applications of learning to rank algorithms using human labels for the training and transductive learning using all the queries available.
Article
In this paper, a new hybrid intrusion detection method that hierarchically integrates a misuse detection model and an anomaly detection model in a decomposition structure is proposed. First, a misuse detection model is built based on the C4.5 decision tree algorithm and then the normal training data is decomposed into smaller subsets using the model. Next, multiple one-class SVM models are created for the decomposed subsets. As a result, each anomaly detection model does not only use the known attack information indirectly, but also builds the profiles of normal behavior very precisely. The proposed hybrid intrusion detection method was evaluated by conducting experiments with the NSL-KDD data set, which is a modified version of well-known KDD Cup 99 data set. The experimental results demonstrate that the proposed method is better than the conventional methods in terms of the detection rate for both unknown and known attacks while it maintains a low false positive rate. In addition, the proposed method significantly reduces the high time complexity of the training and testing processes. Experimentally, the training and testing time of the anomaly detection model is shown to be only 50% and 60%, respectively, of the time required for the conventional models.
Article
Decisions involve many intangibles that need to be traded off. To do that, they have to be measured along side tangibles whose measurements must also be evaluated as to, how well, they serve the objectives of the decision maker. The Analytic Hierarchy Process (AHP) is a theory of measurement through pairwise comparisons and relies on the judgements of experts to derive priority scales. It is these scales that measure intangibles in relative terms. The comparisons are made using a scale of absolute judgements that represents, how much more, one element dominates another with respect to a given attribute. The judgements may be inconsistent, and how to measure inconsistency and improve the judgements, when possible to obtain better consistency is a concern of the AHP. The derived priority scales are synthesised by multiplying them by the priority of their parent nodes and adding for all such nodes. An illustration is included.
Book
The explosive growth of e-commerce and online environments has made the issue of information search and selection increasingly serious; users are overloaded by options to consider and they may not have the time or knowledge to personally evaluate these options. Recommender systems have proven to be a valuable way for online users to cope with the information overload and have become one of the most powerful and popular tools in electronic commerce. Correspondingly, various techniques for recommendation generation have been proposed. During the last decade, many of them have also been successfully deployed in commercial environments. Recommender Systems Handbook, an edited volume, is a multi-disciplinary effort that involves world-wide experts from diverse fields, such as artificial intelligence, human computer interaction, information technology, data mining, statistics, adaptive user interfaces, decision support systems, marketing, and consumer behavior. Theoreticians and practitioners from these fields continually seek techniques for more efficient, cost-effective and accurate recommender systems. This handbook aims to impose a degree of order on this diversity, by presenting a coherent and unified repository of recommender systems major concepts, theories, methodologies, trends, challenges and applications. Extensive artificial applications, a variety of real-world applications, and detailed case studies are included. Recommender Systems Handbook illustrates how this technology can support the user in decision-making, planning and purchasing processes. It works for well known corporations such as Amazon, Google, Microsoft and AT&T. This handbook is suitable for researchers and advanced-level students in computer science as a reference.
Article
Decisions involve many intangibles that need to be traded off. To do that, they have to be measured along side tangibles whose measurements must also be evaluated as to, how well, they serve the objectives of the decision maker. The Analytic Hierarchy Process (AHP) is a theory of measurement through pairwise comparisons and relies on the judgements of experts to derive priority scales. It is these scales that measure intangibles in relative terms. The comparisons are made using a scale of absolute judgements that represents, how much more, one element dominates another with respect to a given attribute. The judgements may be inconsistent, and how to measure inconsistency and improve the judgements, when possible to obtain better consistency is a concern of the AHP. The derived priority scales are synthesised by multiplying them by the priority of their parent nodes and adding for all such nodes. An illustration is included.. He is internationally recognised for his decision-making process, the Analytic Hierarchy Process (AHP) and its generalisation to network decisions, the Analytic Network Process (ANP). He won the Gold Medal from the International Society for Multicriteria Decision Making for his contributions to this field. His work is in decision making, planning, conflict resolution and in neural synthesis.
Article
Prognostics and systems health management (PHM) is an enabling discipline of technologies and methods with the potential of solving reliability problems that have been manifested due to complexities in design, manufacturing, environmental and operational use conditions, and maintenance. Over the past decade, research has been conducted in PHM of information and electronics-rich systems as a means to provide advance warnings of failure, enable forecasted maintenance, improve system qualification, extend system life, and diagnose intermittent failures that can lead to field failure returns exhibiting no-fault-found symptoms.This paper presents an assessment of the state of practice in prognostics and health management of information and electronics-rich systems. While there are two general methods of performing PHM—model-based and data-driven methods—these methods by themselves have some key disadvantages. This paper presents a fusion prognostics approach, which combines or “fuses together” the model-based and data-driven approaches, to enable markedly better prognosis of remaining useful life. A case study of a printed circuit card assembly is given in order to illustrate the implementation of the fusion approach to prognostics.
Conference Paper
In this chapter, we give a brief introduction to learning to rank for information retrieval. Specifically, we first introduce the ranking problem by taking document retrieval as an example. Second, conventional ranking models proposed in the literature of information retrieval are reviewed, and widely used evaluation measures for ranking are mentioned. Third, the motivation of using machine learning technology to solve the problem of ranking is given, and existing learning-to-rank algorithms are categorized and briefly depicted.
Conference Paper
Outlier detection is a data analysis method and has been used to detect and remove anomalous observations from data. In this paper, we firstly introduced some current mainstream outlier detection methodologies, i.e. statistical-based, distance-based, and density-based. Especially, we analyzed distance-based approach and reviewed several kinds of peculiarity factors in detail. Then, we introduced sampled peculiarity factor (SPF) and a SPF-based outlier detection algorithm in order to explore a lower-computational complexity approach to compute peculiarity factor for real world needs in our future work.
Article
This tutorial is concerned with a comprehensive introduction to the research area of learning to rank for information retrieval. In the first part of the tutorial, we will introduce three major approaches to learning to rank, i.e., the pointwise, pairwise, and listwise approaches, analyze the relationship between the loss functions used in these approaches and the widely-used IR evaluation measures, evaluate the performance of these approaches on the LETOR benchmark datasets, and demonstrate how to use these approaches to solve real ranking applications. In the second part of the tutorial, we will discuss some advanced topics regarding learning to rank, such as relational ranking, diverse ranking, semi-supervised ranking, transfer ranking, query-dependent ranking, and training data preprocessing. In the third part, we will briefly mention the recent advances on statistical learning theory for ranking, which explain the generalization ability and statistical consistency of different ranking methods. In the last part, we will conclude the tutorial and show several future research directions.
Article
This paper presents an approach to automatically optimizing the retrieval quality of search engines using clickthrough data. Intuitively, a good information retrieval system should present relevant documents high in the ranking, with less relevant documents following below. While previous approaches to learning retrieval functions from examples exist, they typically require training data generated from relevance judgments by experts. This makes them di#cult and expensive to apply. The goal of this paper is to develop a method that utilizes clickthrough data for training, namely the query-log of the search engine in connection with the log of links the users clicked on in the presented ranking. Such clickthrough data is available in abundance and can be recorded at very low cost. Taking a Support Vector Machine (SVM) approach, this paper presents a method for learning retrieval functions. From a theoretical perspective, this method is shown to be well-founded in a risk minimization framework. Furthermore, it is shown to be feasible even for large sets of queries and features. The theoretical results are verified in a controlled experiment. It shows that the method can e#ectively adapt the retrieval function of a meta-search engine to a particular group of users, outperforming Google in terms of retrieval quality after only a couple of hundred training examples.
AI2: Training a big data machine to defend. Veeramachaneni K. and Arnaldo I. AI2: Training a big data machine to defend
• K Veeramachaneni
• I Arnaldo