Table 5 - uploaded by Sanjay Chawla
Content may be subject to copyright.
The results based on the meta-attributes generated via 1 regression-based landmarker. The remaining 5 accuracy measurements were directly evaluated.
Similar publications
Citations
... In future work, we plan to investigate more sophisticated strategies for building dynamic ensembles such as staking [25] and arbitrated ensembles [18], and also meta-learning for ensemble member selection and ranking [26,27]. Another direction for future work is studying seasonal differences [28] and building ensembles that are better tuned to the seasonal variations. ...
... RMSE) as meta-features for the dataset. An analysis of landmarkers for regression problems can be found in Ler et al. (2005). ...
We investigate the learning of quantitative structure activity relationships (QSARs) as a case-study of meta-learning. This application area is of the highest societal importance, as it is a key step in the development of new medicines. The standard QSAR learning problem is: given a target (usually a protein) and a set of chemical compounds (small molecules) with associated bioactivities (e.g. inhibition of the target), learn a predictive mapping from molecular representation to activity. Although almost every type of machine learning method has been applied to QSAR learning there is no agreed single best way of learning QSARs, and therefore the problem area is well-suited to meta-learning. We first carried out the most comprehensive ever comparison of machine learning methods for QSAR learning: 18 regression methods, 6 molecular representations, applied to more than 2,700 QSAR problems. (These results have been made publicly available on OpenML and represent a valuable resource for testing novel meta-learning methods.) We then investigated the utility of algorithm selection for QSAR problems. We found that this meta-learning approach outperformed the best individual QSAR learning method (random forests using a molecular fingerprint representation) by up to 13%, on average. We conclude that meta-learning outperforms base-learning methods for QSAR learning, and as this investigation is one of the most extensive ever comparisons of base and meta-learning methods ever made, it provides evidence for the general effectiveness of meta-learning over base-learning.
... However, none has investigated the merits of landmarkers as metafeatures. Since these metafeatures use simple estimates of performance to predict the actual performance of algorithms, its efficacy in solving the algorithm selection problem is not only expected but has been demonstrated in various other tasks [3,11,17,18,20,21,25]. Therefore, it is important to understand if their effect is similarly positive in selecting CF algorithms. ...
... Such metafeatures rely on the assumption that by estimating the performance of fast and simple models or by using samples of the data, the performance estimates will correlate well with the best algorithms, hence enabling future predictions. In fact, these metafeatures have proven successful on the selection of algorithms for various tasks [3,11,17,18,20,21,25]. ...
... This section presents our proposal of subsampling landmarkers for the selection of CF algorithms and the experimental procedure used to validate them. Our motivation for using landmarkers is that, although they have been successfully applied to the algorithm selection problem in other learning tasks [3,11,17,18,20,21,25], they were never adapted for selecting CF algorithms. Since there are no fast/simple CF algorithms, which can be used as traditional landmarkers, we have followed the alternative approach of developing subsampling landmarkers, i.e. applying the complete CF algorithms on samples of the data. ...
Recommender Systems have become increasingly popular, propelling the emergence of several algorithms. As the number of algorithms grows, the selection of the most suitable algorithm for a new task becomes more complex. The development of new Recommender Systems would benefit from tools to support the selection of the most suitable algorithm. Metalearning has been used for similar purposes in other tasks, such as classification and regression. It learns predictive models to map characteristics of a dataset with the predictive performance obtained by a set of algorithms. For such, different types of characteristics have been proposed: statistical and/or information-theoretical, model-based and landmarkers. Recent studies argue that landmarkers are successful in selecting algorithms for different tasks. We propose a set of landmarkers for a Metalearning approach to the selection of Collaborative Filtering algorithms. The performance is compared with a state of the art systematic metafeatures approach using statistical and/or information-theoretical metafeatures. The results show that the metalevel accuracy performance using landmarkers is not statistically significantly better than the metafeatures obtained with a more traditional approach. Furthermore, the baselevel results obtained with the algorithms recommended using landmarkers are worse than the ones obtained with the other metafeatures. In summary, our results show that, contrary to the results obtained in other tasks, these landmarkers are not necessarily the best metafeatures for algorithm selection in Collaborative Filtering.
... RMSE) as meta-features for the dataset. An analysis of landmarkers for regression problems can be found in Ler et al. (2005). ...
We investigate the learning of quantitative structure activity relationships (QSARs) as a case-study of meta-learning. This application area is of the highest societal importance, as it is a key step in the development of new medicines. The standard QSAR learning problem is: given a target (usually a protein) and a set of chemical compounds (small molecules) with associated bioactivities (e.g. inhibition of the target), learn a predictive mapping from molecular representation to activity. Although almost every type of machine learning method has been applied to QSAR learning there is no agreed single best way of learning QSARs, and therefore the problem area is well-suited to meta-learning. We first carried out the most comprehensive ever comparison of machine learning methods for QSAR learning: 18 regression methods, 6 molecular representations, applied to more than 2,700 QSAR problems. (These results have been made publicly available on OpenML and represent a valuable resource for testing novel meta-learning methods.) We then investigated the utility of algorithm selection for QSAR problems. We found that this meta-learning approach outperformed the best individual QSAR learning method (random forests using a molecular fingerprint representation) by up to 13%, on average. We conclude that meta-learning outperforms base-learning methods for QSAR learning, and as this investigation is one of the most extensive ever comparisons of base and meta-learning methods ever made, it provides evidence for the general effectiveness of meta-learning over base-learning.
... We will also investigate if feature selection [22] applied to both power and weather data can improve the results. Another direction for future work is selecting the best prediction algorithm for a given solar dataset or different days and times of the day, by investigating methods based on meta-learning [23]. ...
We consider the task of forecasting the electricity power generated by a photovoltaic solar system, for the next day at half-hourly intervals. The forecasts are based on previous power output and weather data, and weather prediction for the next day. We present a new approach that forecasts all the power outputs for the next day simultaneously. It builds separate prediction models for different types of days, where these types are determined using clustering of weather patterns. As prediction models it uses ensembles of neural networks, trained to predict the power output for a given day based on the weather data. We evaluate the performance of our approach using Australian photovoltaic solar data for two years. The results showed that our approach obtained MAE=83.90 kW and MRE=6.88%, outperforming four other methods used for comparison.
... Another interesting extension would be to investigate the application of adaptive wavelet packets with best basis selection -finding the best wavelet basis for different data segments as opposed to finding the best wavelet basis for the whole dataset as in AWNN. Another avenue for future work is algorithm selectionour approach is not limited to using NN as a prediction algorithm and methods for selecting the most appropriate prediction algorithm such as landmarking [43] can be explored. ...
... Relative landmarkers ) remedy this by using performance measures that are predictors of relative, rather than individual performance. While the former studies have focussed on classification, landmarkers for regression have also been proposed (Ler et al. 2005). ...
... Alternatively we could assume that the evaluation should be carried out with respect to the top level in the predicted ranking and the aim is to determine how far down in the correct ranking are the predicted elements. A measure of this type, referred to as average rank of predicted algorithm, was used in [12]. ...
Currently many classification algorithms exist and there is no algorithm that would outperform all the others in all tasks. Therefore it is of interest to determine which classification algorithm is the best one for a given task. Although direct comparisons can be made for any given problem using a cross-validation evaluation, it is desirable to avoid this, as the computational costs are significant. We describe a method which relies on relatively fast pairwise comparisons involving two algorithms. This method exploits sampling landmarks, that is information about learning curves besides classical data characteristics. One key feature of this method is an iterative procedure for extending the series of experiments used to gather new information in the form of sampling landmarks. Metalearning plays also a vital role. The comparisons between various pairs of algorithm are repeated and the result is represented in the form of a partially ordered ranking. Evaluation is done by comparing the partial order of algorithm that has been predicted to the partial order representing the supposedly correct result. The results of our analysis show that the method has good performance and could be of help in practical applications.
... However, the meta-learning community has also developed many other approaches to learning about learning algorithm performance of relevance to classification problems. These include dynamically adjusting the inherent bias in a learning model [meta-learning of broader data mining processes that include algorithm selection and other tasks [Prodromidis et al. 2000; Bernstein et al. 2005]; and landmarking instead of using features for dataset characterization [Pfahringer et al. 2000; Ler et al. 2005]. All of these tasks are called meta-learning because of their focus on learning about learning algorithms. ...
... Beyond the Algorithm Selection Problem, there are many ideas under the banner of meta-learning that can be adopted by researchers from other disciplines. While it is beyond the scope of the current paper to review these methods, the interested reader may follow up on some of the concepts that could find generalisation beyond machine learning, including: combining algorithms and voting schemes (boosting, bagging, stacked generalisation, mixture of experts modelling – see for example Wolpert [1992]; Chan and Stolfo [1997]; Opitz and Maclin [1999]; Gama and Brazdil [2000]; Todorovski and Dzeroski [2003]; Peterson and Martinez [2005]), algorithm portfolios [Gagliolo and Schmidhuber 2006], landmarking [Pfahringer et al. 2000; Ler et al. 2005], dynamic algorithm selection [Armstrong et al. 2006; Samulowitz and Memisevic 2007] and real-time analysis of algorithms, particularly dynamic tuning of parameters via racing algorithms [Maron and Moore 1997]. Similarly, there are many ideas developed in the AI community that could be worth generalising to other disciplines. ...
The algorithm selection problem [Rice 1976] seeks to answer the question: Which algorithm is likely to perform best for my problem? Recognizing the problem as a learning task in the early 1990's, the machine learning community has developed the field of meta-learning, focused on learning about learning algorithm performance on classification problems. But there has been only limited generalization of these ideas beyond classification, and many related attempts have been made in other disciplines (such as AI and operations research) to tackle the algorithm selection problem in different ways, introducing different terminology, and overlooking the similarities of approaches. In this sense, there is much to be gained from a greater awareness of developments in meta-learning, and how these ideas can be generalized to learn about the behaviors of other (nonlearning) algorithms. In this article we present a unified framework for considering the algorithm selection problem as a learning problem, and use this framework to tie together the crossdisciplinary developments in tackling the algorithm selection problem. We discuss the generalization of meta-learning concepts to algorithms focused on tasks including sorting, forecasting, constraint satisfaction, and optimization, and the extension of these ideas to bioinformatics, cryptography, and other fields.
... Muslea et al. [11] described an approach that attempts to learn if two views are sufficiently compatible in multi-view learning for the tasks of wrapper induction and text classification. In the area of algorithm selection, we have developed an approach which, for a given task, selects the best classifier from a set of classifiers, given their previous performance on other problems [12]. We plan to extend these approaches to predict if a task is suitable for cotraining , and what the co-training parameters should be. ...
In this paper we present a case study of co-training to image classification. We consider two scene classification tasks: indoors vs. outdoors and animals vs. sports. The results show that co-training with Naïve Bayes using 8-10 labelled examples obtained only 1.2-1.5% lower classification accu-racy than Naïve Bayes trained on the full labelled version of the training set (138 examples in task 1 and 827 examples in task 2). Co-training was found to be sensitive to the choice of base classifier, with Naïve Bayes outperforming Random Forest. We also propose a simple co-training modification based on the different inductive basis of classification algo-rithms and show that it is a promising approach.
