Conference Paper

Distributed scheduling in large scale monitoring infrastructures.

DOI: 10.1145/1544012.1544065 Conference: Proceedings of the 2008 ACM Conference on Emerging Network Experiment and Technology, CoNEXT 2008, Madrid, Spain, December 9-12, 2008
Source: DBLP

ABSTRACT Network monitoring is becoming a necessity for network operators, who usually deploy several monitoring applications that aid in tasks such as traffic engineering, capacity planning and the detection of attacks or other anomalies. There is also an increasing interest in large-scale network monitoring infrastructures that can run multiple applications in several network viewpoints [4].

Download full-text


Available from: Pere Barlet-Ros, Sep 28, 2015
22 Reads
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Monitoring and mining real-time network data streams is crucial for managing and operating data networks. The information that network operators desire to extract from the network traffic is of different size, granularity and accuracy depending on the measurement task (e.g., relevant data for capacity planning and intrusion detection are very different). To satisfy these different demands, a new class of monitoring systems is emerging to handle multiple arbitrary and continuous traffic queries. Such systems must cope with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic. In this paper, we present the design and evaluation of a system that can shed excess load in the presence of extreme traffic conditions, while maintaining the accuracy of the traffic queries within acceptable levels. The main novelty of our approach is that it is able to operate without explicit knowledge of the traffic queries. Instead, it extracts a set of features from the traffic streams to build an on-line predictionmodel of the query resource requirements. This way the monitoring system preserves a high degree of flexibility, increasing the range of applications and network scenarios where it can be used. We implemented our scheme in an existing network monitoring system and deployed it in a research ISP network. Our results show that the system predicts the resources required to run each traffic query with errors below 5%, and that it can efficiently handle extreme load situations, preventing uncontrolled packet losses, with minimum impact on the accuracy of the queries' results.
    Proceedings of the 2007 USENIX Annual Technical Conference, June 17-22, 2007, Santa Clara, CA, USA; 01/2007
  • [Show abstract] [Hide abstract]
    ABSTRACT: Developing new tools to analyze network data is often a com- plex and error-prone process. Current practices require developers to pos- sess an in-depth understanding of the original data sets and to develop ad-hoc software tools to first extract the relevant information from the data and then implement the internals of the new algorithm. This de- velopment process results in long delays during the analysis of the data and in the production of software that is often hard to reuse, debug or optimize. We present the design and implementation of CoMo, a system for fast prototyping network data mining applications. CoMo provides an abstraction layer both for the network data as well as for the hardware architecture used to collect and process the data. This allows developers to focus on the correctness of the implementation of their analysis tools while the system makes the tool amenable to optimization when running on dierent hardware architectures. In this paper we discuss CoMo's de- sign challenges, our solution to address them and evaluate the resulting software in terms of performance, flexibility and ease of use.
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: One measure of the usefulness of a general-purpose distributed computing system is the system's ability to provide a level of performance commensurate to the degree of multiplicity of resources present in the system. A taxonomy of approaches to the resource management problem is presented in an attempt to provide a common terminology and classification mechanism necessary in addressing this problem. The taxonomy, while presented and discussed in terms of distributed scheduling, is also applicable to most types of resource management
    IEEE Transactions on Software Engineering 03/1988; 14(2-14):141 - 154. DOI:10.1109/32.4634 · 1.61 Impact Factor