No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Machine learning for measurement-based bandwidth estimation
Abstract
The dispersion that arises when packets traverse a network carries information that can reveal relevant network characteristics. Using a fluid-flow model of a bottleneck link with first-in first-out multiplexing, accepted probing tools measure the packet dispersion to estimate the available bandwidth, i.e., the residual capacity that is left over by other traffic. Difficulties arise, however, if the dispersion is distorted compared to the model, e.g., by non-fluid traffic, multiple bottlenecks, clustering of packets due to interrupt coalescing, and inaccurate time-stamping in general. It is recognized that modeling these effects is cumbersome if not intractable. This motivates us to explore the use of machine learning in bandwidth estimation. We train a neural network using vectors of the packet dispersion that is characteristic of the available bandwidth. Our testing results reveal that even a shallow neural network identifies the available bandwidth with high precision. We also apply the neural network under a variety of notoriously difficult conditions that have not been included in the training, such as randomly generated networks with the multiple bottleneck links and heavy cross traffic burstiness. Compared to two state-of-the-art model-based techniques as well as a recent machine learning-based technique (Yin et al., 2016), our neural network approach shows improved performance. Further, our neural network can effectively control the estimation procedure in an iterative implementation. We also evaluate our method with other supervised machine learning techniques.
... It is also important to note that while deploying an admission control scheme, focus should not be directed on only the gateway for QoS to be maintained in MANET. There must be adequate provision for locations that are hotspot centred, especially those locations around the MAC layer [8]. All these requirements were considered during the designing of our proposed admission control algorithm to achieve a good and guaranteed QoS in MANET. ...
... Bandwidth capacity is the amount of traffic that can be transmitted by a network without causing either a traffic degradation or affecting other flows transmitting within the network [13]. When the capacity of the channel is accurately determined, it ensures an easy deployment of the admission control scheme on the network to prevent network congestion within MANET [8]. Due to the fact that mobile nodes share the wireless network medium based on contention, there is a relationship between the capacity of a channel and the bandwidth of the channel. ...
... The channels busyness ratio in MANET and the current loss rate. The former emulates a TCP connection by measuring the actual throughput within the network, while the latter estimates the current loss rate, thereby deriving a bandwidth estimation rate from the lost packet [8], [14], [15]. ...
The widespread deployment of mobile ad-hoc network (MANET) in the areas of agriculture, military defence, weather forecasting and disaster control has necessitated the implementation of admission control within a network for a guaranteed quality of service (QoS). Admission control organises traffic flows to ensure the network medium is fairly shared among various nodes in the network. Various admission control algorithms have been proposed in the literature, using different metrics and parameters to achieve different admission control quality. In this work, we propose an Admission control in mobile ad-hoc network routing (ACMANR) using both bandwidth capacity and resource estimation to achieve a good QoS. Furthermore, we analyse the behaviour of two well-known routing protocols in wireless network, ad hoc on-demand distance vector (AODV) and dynamic source routing (DSR), in our proposed admission control algorithm. Simulation results obtained from our proposed admission control algorithm using OPNET show that AODV routing protocol had a better throughput while DSR produced a better delay with lower overhead in MANET. Our proposed approach also shows better performance in terms of throughput and delay when compared with the state-of-the-art admission control routing using AODV and DSR.
... 3) Multiple Tight Links: We extend our testbed from the single-hop network to the multi-hop network, as shown in Fig. 2 Fig. 9. A two-hop network where the tight link differs from the bottleneck link [30]. ...
... (a) Rate response curves[30]200 400 600 800 1000 Steps 0 20 40 60 80 100 Available Bandwidth Estimates (Mbps) True available bandwidth Direct probing Reinforcement learning Direct probing (SD) Reinforcement learning (SD) (b) Available bandwidth estimates for scenario I 200 400 600 800 1000 Steps 0 20 40 60 80 100 Available Bandwidth Estimates (Mbps) True available bandwidth Direct probing Reinforcement learning Direct probing (SD) Reinforcement learning (SD) (c) Available bandwidth estimates for scenario II ...
An accurate and fast estimation of the available bandwidth in a network with varying cross-traffic is a challenging task. The accepted probing tools, based on the fluid-flow model of a bottleneck link with first-in, first-out multiplexing, estimate the available bandwidth by measuring packet dispersions. The estimation becomes more difficult if packet dispersions deviate from the assumptions of the fluid-flow model in the presence of non-fluid bursty cross-traffic, multiple bottleneck links, and inaccurate time-stamping. This motivates us to explore the use of machine learning tools for available bandwidth estimation. Hence, we consider reinforcement learning and implement the single-state multi-armed bandit technique, which follows the -greedy algorithm to find the available bandwidth. Our measurements and tests reveal that our proposed method identifies the available bandwidth with high precision. Furthermore, our method converges to the available bandwidth under a variety of notoriously difficult conditions, such as heavy traffic burstiness, different cross-traffic intensities, multiple bottleneck links, and in networks where the tight link and the bottleneck link are not same. Compared to the piece-wise linear network a model-based direct probing technique that employs a Kalman filter, our method shows more accurate estimates and faster convergence in certain network scenarios and does not require measurement noise statistics.
... The LSTM shows better performance than the other two approaches by achieving only 3% error rate. Khangura et al. 43 estimated the available bandwidth using a shallow neural network. They used vectors of packet dispersion as input features, which are characteristic of the available bandwidth. ...
The Fifth Generation (5G) of mobile communications is the most exciting emerging technology for researchers and scientists to get the full benefit of a network system. However, 5G networks confront massive threats and vulnerabilities including protection, privacy, and secrecy. To face these challenges in the increasingly interconnected Internet of Things (IoT) scenario, we aim to leverage state-of-the-art technologies as Software Defined Networking (SDN) in conjunction with Network Function Virtualization (NFV), Blockchain, and Machine Learning (ML). Indeed, these technologies convey a robust and secure setting in the networking platform enabling to manage several criticalities related to security, privacy, flexibility, and performance. In light of these considerations, in this paper, we propose the "BlockSD-5GNet" architecture to efficiently improve the security of a 5G network and to exploit the combined advantages of Blockchain, SDN, NFV, and ML. In the proposed architecture, the SDN helps to manage the network by dividing it into data plane and control plane, while the Blockchain guarantees improved security and confidentiality. Therefore, the "BlockSD-5GNet" architecture can both secure sensitive data and attain reliable data transfer within and between the 5G network-infrastructure planes. Additionally, an ML module is integrated into the SDN controller to estimate network bandwidth and assist the administrator in taking effective decisions and satisfying high-bandwidth demand. We assess the performance of the "BlockSD-5GNet" architecture via an experimental evaluation performed in a simulation environment, and show the effectiveness of the proposed solution in comparison with baseline schemes. Finally, we also demonstrate the capability of different ML models in bandwidth prediction.
... It took into account network conditions, the financial cost of the network, the QoS requirements of the applications, user preferences, and the energy consumption of the mobile device. The research paper [17] concentrated on the potential use of machine learning-one of the uses for ANNs and a component of them-in wireless communications. The basic architecture, training process, difficulties, and opportunities associated with the various types of NNs, such as feedforward, recurrent, and deep learning NNs, were reviewed. ...
Given the variety of networks, interfaces, mediums, and other resources accessible in a wireless heterogeneous location today, mobile communication is in a healthy competitive environment. When clients or users have access to numerous interfaces at once, a problem occurs. Users therefore require a clever or intelligent system to connect them to the finest services based on their needs and choices. Interface management controls the interfaces that are accessible and links the user to the best. The intelligent usage of various radio accesses/interfaces is made possible in this research by interface management with artificial neural networks (ANNs). Various parameters of different networks are used to make the choice. In this paper, a back propagation neural network (BPNN) method for switching between 3G, WLAN, 4G, and 5G networks is suggested. By allocating appropriate weights, the various network properties are employed as selection parameters. Fuzzy Analytic Hierarchy Process (FAHP) is used to initialize the weights, and the BPNN is used to optimize them. In order to obtain the best value, the goal value and the original value are compared, and the difference between them is used as an adjusting value for the weights. The network is trained using backpropagation techniques. The comparison between the proposed algorithm and the current algorithm demonstrates the new method's flexibility and the network's optimal connectivity with a high rate of successful handovers.
... In contrast to model-driven approaches, data-driven approaches employ machine-learning methods to identify and learn relationships between latency and other variables using the measurement data from real systems. Khangura et al. identified bottleneck links and estimated the residual bandwidth via a neural network that was trained using a vector of packet dispersion values [7]. The output of the estimator produced a point estimate of the performance metric (i.e., available bandwidth). ...
With the emergence of new application areas, such as cyber-physical systems and human-in-the-loop applications, there is a need to guarantee a certain level of end-to-end network latency with extremely high reliability, e.g., 99.999%. While mechanisms specified under IEEE 802.1as time-sensitive networking (TSN) can be used to achieve these requirements for switched Ethernet networks, implementing TSN mechanisms in wireless networks is challenging due to their stochastic nature. To conform the wireless link to a reliability level of 99.999%, the behavior of extremely rare outliers in the latency probability distribution, or the tail of the distribution, must be analyzed and controlled. This work proposes predicting the tail of the latency distribution using state-of-the-art data-driven approaches, such as mixture density networks (MDN) and extreme value mixture models, to estimate the likelihood of rare latencies conditioned on the network parameters, which can be used to make more informed decisions in wireless transmission. Actual latency measurements of IEEE 802.11g (WiFi), commercial private and a software-defined 5G network are used to benchmark the proposed approaches and evaluate their sensitivities concerning the tail probabilities.
... The different types of neural networks such as feedforward, recurrent, and deep learning neural networks their basic architecture, training procedure, challenges and opportunities related to them were overviewed. The neural network approach was used to estimate the bandwidth [14]. To minimize the cross-layer interference of multiple base station cells, a cluster-based cooperative interference management scheme for a multi-interface environment was proposed in [15]. ...
In today’s scenario, mobile communication is facing a healthy competition due to different networks, interfaces, channels, and many more available in wireless heterogeneous environment. The problem arises when customers/users get the availability of many interfaces at the same time. At that time users need an intelligent or smart mechanism to connect them to the best services according to their requirements/preferences. Interface management manages available interfaces and connects the user with the best. In this paper, Interface management with Artificial Neural Network (ANN) allows the smart use of different radio accesses/interfaces. The selection is made with different parameters of different networks. This paper proposed a backpropagation neural network that is used for the switching in between different networks-3G, WLAN, 4G and 5G. The different parameters of a network are used as the selection parameters with assigning proper weights. Weights are initialized with fuzzy AHP and optimized with Back Propagation Neural Network (BPNN). The target value and the actual value is compared and their difference used as the adjusting value for the weights to get the optimum value. The backpropagation is used to train the network. The comparison among the projected algorithm and the existing algorithm shows the valuablity of the new method and the best connectivity of the network.
... Measurements are collected during the connection is active or also gathered by active or passive measurements before the actual connection starts. Also, the training may occur supervised on historical data or unsupervised during the connection is active, typically by optimization of a utility function.Machine learning applied to the estimation of available bandwidth estimation is explored in [23], [24], [25]. The authors show the higher accuracy achieved by supervised machine learning, which leads to a reduction of probing traffic if compared to conventional estimation tools. ...
Selection of the optimal transmission rate in packet-switched best-effort networks is challenging. Typically, senders do not have any information about the end-to-end path and should not congest the connection but at once fully utilize it. The accomplishment of these goals lead to congestion control protocols such as TCP Reno, TCP Cubic, or TCP BBR that adapt the sending rate according to extensive measurements of the path characteristics by monitoring packets and related acknowledgments. To improve and speed up this adaptation, we propose and evaluate a machine learning approach for the prediction of sending rates from measurements of metrics provided by the TCP stack. For the prediction a neural network is trained and evaluated. The prediction is implemented in the TCP stack to speed up TCP slow start. For a customizable and performant implementation the extended Berkeley packet filter is used to extract relevant data from the kernel space TCP stack, to forward the monitoring data to a user space data rate prediction, and to feed the prediction result back to the stack. Results from a online experiment show improvement in flow completion time of up to 30%.
... In Khangura et al. [2019], Yin and Kaur [2016] the authors train a neural network using vectors constructed by packets. The vectors contain the available bandwidth of the packet dispersion. ...
Real-Time Networks (RTNs) provide latency guarantees for time-critical applications and it aims to support different traffic categories via various scheduling mechanisms. Those scheduling mechanisms rely on a precise network performance measurement to dynamically adjust the scheduling strategies. Machine Learning (ML) offers an iterative procedure to measure network performance. Network Calculus (NC) can calculate the bounds for the main performance indexes such as latencies and throughputs in an RTN for ML. Thus, the ML and NC integration improve overall calculation efficiency. This paper will provide a survey for different approaches of Real-Time Network performance measurement via NC as well as ML and present their results, dependencies, and application scenarios.
... In [10], the active available bandwidth estimation technique is classified into isolated probing, direct probing, and iterative probing. [11] classified it into direct probing and iterative probing technique. In [12], the active available bandwidth estimation was classified into direct probing, iterative probing, and mixed techniques while [13] classified it into packet dispersion measurement (PDM), probe gap model (PGM) and probe rate model (PRM). ...
In wireless network, the support for Quality of Service (QoS) depends on the ability of a network to estimate the available bandwidth of a link effectively and efficiently. Due to the shared nature of the wireless medium, there has been challenges in achieving an accurate measurement of the available bandwidth. This paper therefore proposes an efficient available bandwidth estimation approach that improves the accuracy of previously proposed approach for accurate measurement. The approach used in this paper is to passively estimate the available bandwidth, where the maximum range of the channel idle time measurement used in the calculation is re-defined to address the outer zone range threshold (OZR). This limits the under-estimation and over-estimation of bandwidth measurement. The proposed approach used limits the network overhead and achieves accuracy and efficiency. Results obtained from the simulation shows how our proposed approach out-performs the state-of the-art approach in terms of throughput and delay, using the IEEE802.11e. We have been able to achieve 8% improvement in accuracy and efficiency when compared with the best of the state-of-the-art available bandwidth estimation technique.
... By load-or interchangeably, traffic demands-(S ), we can refer, simply, the bandwidth-usage time series in absolute terms-i.e., Mb/s-or normalized by capacity to make equally relevant each PoP of a path-i.e., in percentage of use. Alternatively, such traffic-demand time series can be the complementary of the available bandwidth per PoP-i.e., the difference between capacity and used bandwidth [40,41]. In such a way, the valley time of two PoPs would be that moment when more aggregate bandwidth is available instead of the moment with less bandwidth in use. ...
Periods of low load have been used for the scheduling of non-interactive tasks since the early stages of computing. Nowadays, the scheduling of bulk transfers-i.e., large-volume transfers without precise timing, such as database distribution, resources replication or backups-stands out among such tasks, given its direct effect on both the performance and billing of networks. Through visual inspection of traffic-demand curves of diverse points of presence (PoP), either a network, link, Internet service provider or Internet exchange point, it becomes apparent that low-use periods of bandwidth demands occur at early morning, showing a noticeable convex shape. Such observation led us to study and model the time when such demands reach their minimum, on what we have named valley time of a PoP, as an approximation to the ideal moment to carry out bulk transfers. After studying and modeling single-PoP scenarios both temporally and spatially seeking homogeneity in the phenomenon, as well as its extension to multi-PoP scenarios or paths-a meta-PoP constructed as the aggregation of several single PoPs-, we propose a final predictor system for the valley time. This tool works as an oracle for scheduling bulk transfers, with different versions according to time scales and the desired trade-off between precision and complexity. The evaluation of the system, named VTP, has proven its usefulness with errors below an hour on estimating the occurrence of valley times, as well as errors around 10% in terms of bandwidth between the prediction and actual valley traffic.
... The network status is measured periodically, including the network bandwidth and transmission latency. The available network bandwidth measurement approaches include variable packet size (VPS), packet pair, probing gap model and probing rate model 4 . ...
In this paper, we propose the mobile edge computing based latency aware OCR and realtime english translation (MEC‐Trans) for smart cities. MEC‐Trans monitors the local resource status of the edge platform and the network status from the edge to the cloud in real time. Based on this, it adopts an intelligent decision mechanism driven by user experience. It offloads some computation to the edge and dynamically reduces the network transmission data amount. In this way a relatively stable quality of user experience can be guaranteed.
... We provide data sets generated in a controlled network testbed located at Leibniz University Hannover for training and testing of our proposed method. The work in this chapter is based on joint work with Markus Fidler and Bodo Rosenhahn [40,41]. ...
Today’s Internet Protocol (IP), the Internet’s network-layer protocol, provides
a best-effort service to all users without any guaranteed bandwidth. However,
for certain applications that have stringent network performance requirements
in terms of bandwidth, it is significantly important to provide Quality of Service (QoS) guarantees in IP networks. The end-to-end available bandwidth of a
network path, i.e., the residual capacity that is left over by other traffic, is determined by its tight link, that is the link that has the minimal available bandwidth.
The tight link may differ from the bottleneck link, i.e., the link with the minimal
capacity.
Passive and active measurements are the two fundamental approaches used
to estimate the available bandwidth in IP networks. Unlike passive measurement tools that are based on the non-intrusive monitoring of traffic, active tools
are based on the concept of self-induced congestion. The dispersion, which
arises when packets traverse a network, carries information that can reveal relevant network characteristics. Using a fluid-flow probe gap model of a tight link
with First-in, First-out (FIFO) multiplexing, accepted probing tools measure the
packet dispersion to estimate the available bandwidth. Difficulties arise, however, if the dispersion is distorted compared to the model, e.g., by non-fluid
traffic, multiple tight links, clustering of packets due to interrupt coalescing
and inaccurate time-stamping in general. It is recognized that modeling these
effects is cumbersome if not intractable.
To alleviate the variability of noise-afflicted packet gaps, the state-of-the-art
bandwidth estimation techniques use post-processing of the measurement results, e.g., averaging over several packet pairs or packet trains, linear regression,
or a Kalman filter. These techniques, however, do not overcome the basic assumptions of the deterministic fluid model. While packet trains and statistical
post-processing help to reduce the variability of available bandwidth estimates,
these cannot resolve systematic deviations such as the underestimation bias
in case of random cross traffic and multiple tight links. The limitations of the
state-of-the-art methods motivate us to explore the use of machine learning in
end-to-end active and passive available bandwidth estimation.
We investigate how to benefit from machine learning while using standard packet train probes for active available bandwidth estimation. To reduce
the amount of required training data, we propose a regression-based scaleinvariant method that is applicable without prior calibration to networks of arbitrary capacity. To reduce the amount of probe traffic further, we implemen
a neural network that acts as a recommender and can effectively select the
probe rates that reduce the estimation error most quickly. We also evaluate our
method with other regression-based supervised machine learning techniques.
Furthermore, we propose two different multi-class classification-based methods for available bandwidth estimation. The first method employs reinforcement learning that learns through the network path’s observations withou
having a training phase. We formulate the available bandwidth estimation as a
single-state Markov Decision Process (MDP) multi-armed bandit problem and
implement the "-greedy algorithm to find the available bandwidth, where " is
a parameter that controls the exploration vs. exploitation trade-off.
We propose another supervised learning-based classification method to obtain reliable available bandwidth estimates with a reduced amount of network
overhead in networks, where available bandwidth changes very frequently. In
such networks, reinforcement learning-based method may take longer to converge as it has no training phase and learns in an online manner. We also evaluate our method with different classification-based supervised machine learning techniques. Furthermore, considering the correlated changes in a network’s
traffic through time, we apply filtering techniques on the estimation results in
order to track the available bandwidth changes.
Active probing techniques provide flexibility in designing the input structure. In contrast, the vast majority of Internet traffic is Transmission Contro
Protocol (TCP) flows that exhibit a rather chaotic traffic pattern. We investigate
how the theory of active probing can be used to extract relevant information
from passive TCP measurements. We extend our method to perform the estimation using only sender-side measurements of TCP data and acknowledgmen
packets. However, non-fluid cross traffic, multiple tight links, and packet loss
in the reverse path may alter the spacing of acknowledgments and hence increase the measurement noise. To obtain reliable available bandwidth estimates
from noise-afflicted acknowledgment gaps we propose a neural network-based
method.
We conduct a comprehensive measurement study in a controlled network
testbed at Leibniz University Hannover. We evaluate our proposed methods
under a variety of notoriously difficult network conditions that have not been
included in the training such as randomly generated networks with multiple
tight links, heavy cross traffic burstiness, delays, and packet loss. Our testing
results reveal that our proposed machine learning-based techniques are able to
identify the available bandwidth with high precision from active and passive
measurements. Furthermore, our reinforcement learning-based method without any training phase shows accurate and fast convergence to available bandwidth estimates.
... The variability of the available bandwidth estimates of the direct method is comparably large and the average underestimates the true available bandwidth. Though variability of estimates could be due to a number of reasons [30], particularly in this case, the exponential cross traffic deviates from the fluid model and causes random fluctuations of the measurements of g ack . The neural network-based method improves bandwidth estimates significantly. ...
The information that short-lived TCP flows provide
on bandwidth estimation may benefit adaptive video streaming
applications or may contribute towards the success of new TCP
versions. To estimate the available bandwidth active probing
techniques may be used, but they cause an intrusive effect
on the network affecting TCP performance. Therefore, passive
measurements are favored, though capturing traffic traces comes
with its own challenges. In this paper, we use the feedback
provided by TCP acknowledgments and perform the estimation
from sender-side measurements only. However, difficulties arise
when in the presence of discrete random cross traffic, multiple
tight links, packet losses, and inaccurate timestamping in general,
the acknowledgment packet gaps get distorted. To deal with noiseafflicted packet gaps, we consider a machine-learning approach,
specifically the neural network, for estimating the available
bandwidth. We also apply the neural network under a variety
of notoriously difficult conditions that have not been included
in the training, such as multiple tight links, heavy cross traffic
burstiness and packet losses. We compare the performance
of our proposed method with a state-of-the-art model-based
technique, where our neural network approach shows improved
performance.
... Authors also draw attention towards the pitfalls and difficulties faced by existing available bandwidth estimation criteria in real-time networks. Authors also investigated neural network-based available bandwidth estimation [33]. They considered a vector of ratios of equidistantly spaced probe rates at the sender and at the receiver as input to a neural network to estimate the available bandwidth and the bottleneck capacity. ...
Background
With the availability of multiple options in wireless network simultaneously, Always Best Connected (ABC) requires dynamic selection of the best network and access technologies.
Objective
In this paper, a novel dynamic access network selection algorithm based on the real time is proposed. The available bandwidth (ABW) of each network is required to be estimated to solve the network selection problem.
Method
Proposed algorithm estimates available bandwidth by taking averages, peaks, low points and bootstrap approximation for network selection. It monitors real-time internet connection and resolves the selection issue in internet connection. The proposed algorithm is capable of adapting to prevailing network conditions in heterogeneous environment of 2G, 3G and WLAN networks without user intervention. It is implemented in temporal and spatial domains to check its robustness. Estimation error, overhead, estimation time with the varying size of traffic and reliability are used as the performance metrics.
Results
Through numerical results, it is shown that the proposed algorithm’s ABW estimation based on bootstrap approximation gives improved performance in terms of estimation error (less than 20%), overhead (varies from 0.03% to 83%) and reliability (approx. 99%) with respect to existing techniques.
Conclusion
Our proposed methodology of network selection criterion estimates the available bandwidth by taking averages, peaks, and low points and bootstrap approximation method (standard deviation) for the selection of network in the wireless heterogeneous environment. It monitors real-time internet connection and resolves internet connections selection issue. All the real-time usage and test results demonstrate the productivity and adequacy of available bandwidth estimation with bootstrap approximation as a practical solution for consistent correspondence among heterogeneous wireless networks by precise network selection for multimedia services.
To improve the efficiency of data resource scheduling in virtual data centers, this study analyzes the network structure of the data center, constructs a set of key nodes, and selects the importance of nodes. And a mapping algorithm based on key nodesis proposed as the mapping algorithm for the virtual data center. The effectiveness of virtual data center compression and network feature validation are analyzed. In the DCell structure, after compressing the virtual data center, the request acceptance rate has improved. The compressed virtual data center significantly consumes less physical bandwidth during mapping compared to the case of direct virtual data center mapping. In the BCube structure, after compressing the virtual data center, the request acceptance rate has also improved. A comparative experiment is conducted between the mapping algorithm based on key nodes and the mapping algorithm based ontherandom experiments. Under the three architectures of DCell, BCube, and Fattree, the acceptance rate of key node mapping algorithms is higher than that ofthe random experimental mapping algorithms. Its bandwidth utilization rate is also lower than the mapping algorithm in random experiments. After introducing network features, the request acceptance rate has improved to varying degrees. The bandwidth utilization rate increases with the increase of acceptance rate.
In network management, network measuring is crucial. Accurate network measurements can increase network utilization, network management, and the ability to find network problems promptly. With extensive technological advancements, the difficulty for network measurement is not just the growth in users and traffic but also the increasingly difficult technical problems brought on by the network’s design becoming more complicated. In recent years, network feature measurement issues have been extensively solved by the use of ML approaches, which are ideally suited to thorough data analysis and the investigation of complicated network behavior. However, there is yet no favored learning model that can best address the network measurement issue. The problems that ML applications in the field of network measurement must overcome are discussed in this study, along with an analysis of the current characteristics of ML algorithms in network measurement. Finally, network measurement techniques that have been used as ML techniques are examined, and potential advancements in the field are explored and examined.
Integration of mobility to sensor node enhanced the scalability of Ad-hoc networks. Bandwidth estimation in scenarios where nodes move randomly and frequently changes its link may depend on factors other than congestion in network traffic. The event of link failure due to the high mobility behavior of nodes in the path from source to destination is one of the dominating cause. However, in both cases, TCP’s new-Reno and its variants reduce the congestion-window size to half or 1-MSS, which can cause bandwidth underutilization in the event of a packet drop due to link failure. The proposed approach distinguishes the link failure from network congestion and estimates bandwidth based on link and path stability matrics. The contribution includes the identification of node mobility fuzziness in unicast and broadcast scenarios of IEEE 802.15.4 based MANET. The mobility-fuzziness formulation by the proposed NSGA-II optimized fuzzy-inference-system imitates the node’s mobility behavior and, in the event of packet loss, is employed to realize the path-stability metric to estimate the congestion window size. The thorough evaluation with current state-of-the-art techniques shows that the proposed path stability metric allows the estimation of congestion window size to the current congestion window in the event of a packet loss due to link failure. Improving the bandwidth utilization metric in high mobility scenarios from 53–61% for existing approaches to above 83% in proposed approach.
In order to answer how much bandwidth is available to an application from one end to another in a network, state-of-the-art estimation techniques, based on active probing, inject artificial traffic with a known structure into the network. At the receiving end, the available bandwidth is estimated by measuring the structural changes in the injected traffic, which are caused by the network path. However, bandwidth estimation becomes difficult when packet distributions are distorted by non-fluid bursty cross traffic and multiple links. This eventually leads to an estimation bias. One known approach to reduce the bias in bandwidth estimations is to probe a network with constant-rate packet trains and measure the average structural changes in them. However, one cannot increase the number of packet trains in a designated time period as much as needed because high probing intensity overloads the network and results in packet losses in probe and cross traffic, which distorts probe packet gaps and inflicts more bias. In this work, we propose a machine learning-based, particularly classification-based, method that provides reliable estimates utilizing fewer packet trains. Then, we implement supervised learning techniques. Furthermore, considering the correlated changes over time in traffic in a network, we apply filtering techniques on estimation results in order to track the changes in the available bandwidth. We set up an experimental testbed using the Emulab software and a dumbbell topology in order to create training and testing data for performance analysis. Our results reveal that our proposed method identifies the available bandwidth significantly well in single-link networks as well as networks with heavy cross traffic burstiness and multiple links. It is also able to estimate the available bandwidth in randomly generated networks where the network capacity and the cross traffic intensity vary substantially. We also compare our technique with the others that use direct probing and regression approaches, and show that ours has better performance in terms of standard deviation around the actual bandwidth values.
Wind speed forecasting plays a pivotal role in the security and economy of the power system operation. However, accurate prediction on the wind speed value is still quite challenging due to three features of wind speed: randomness, fluctuation and unpredictability. Though many previous works have studied decomposition-based methods for preprocessing, these methods are difficult to use in actual prediction, since the newly acquired data will greatly affect the values of the initial decomposed subsequences. To solve this problem, a good solution referred as the wavelet soft threshold denoising (WSTD) is proposed herein, which is also systematically introduced for the first time. Moreover, a novel deep learning method gated recurrent unit (GRU) is successfully developed to be combined with WSTD to forecast the wind speed series in this paper. In the ensemble WSTD-GRU model, the WSTD is applied to filter the redundant information from the original data of wind speed series; GRU, which is particularly advantageous for learning the features in case of cooperating with WSTD, is adopted to predict the future multi-step wind speed values. Also, the parameters of the GRU are adjusted by cross-validated grid-search, and they are visualized in the paper. In experiments, thirty-five commonly used models are involved for comparison to evaluate the forecasting performance of the proposed model, through the datasets from four sites in the United States covering a wide range of longitudes. Finally, a superior performance of this methodology with relatively high accuracy, fast forecasting speed, small volatility of results and good adaptability is verified by the test results of case studies aforementioned.
We present MoonGen, a flexible high-speed packet generator. MoonGen is able
to saturate 10 GbE links with minimum sized packets using only a single CPU
core by running on top of the packet processing framework DPDK. We move the
whole packet generation logic into user-controlled Lua scripts, thereby
ensuring high flexibility. MoonGen works with commodity NICs and provides
features that were previously only possible with hardware-based solutions. We
utilize hardware features of Intel NICs that have not been used for packet
generators previously. MoonGen can measure latencies with sub-microsecond
precision and accuracy by using hardware timestamping, a feature that is
available on modern commodity NICs. In the evaluation of MoonGen, we address
timing issues with software-based packet generators and show how they can be
mitigated. We provide MoonGen as free software under the MIT license at
https://github.com/emmericp/MoonGen
The area of available bandwidth (avail-bw) estimation has attracted significant interest recently, with several estimation techniques and tools developed during the last 2-3 years. Unfortunately, some key issues regarding the avail-bw definition, estimation, and validation remain vague or misinterpreted. In this note, we first review the previous work in the area and classify the existing techniques in two classes: direct probing and iterative probing. We then identify ten misconceptions, in the form of fallacies or pitfalls, that we consider as most important. Some misconceptions relate to basic statistics, such as the impact of the population variance on the sample mean, the variability of the avail-bw in different time scales, and the effect of the probing duration. Other misconceptions relate to the queueing model underlying these estimation techniques. For instance, ignoring that traffic burstiness or the presence of multiple bottlenecks can cause significant underestimation errors. Our objective is not to debunk previous work or to claim that some estimation techniques are better than others, but to clarify a number of important issues that cover the entire area of avail-bw estimation so that this important metric can be better understood and put in practical use.
In this paper we present our traffic generation platform, named D-ITG (distributed Internet traffic generator). Its features are first described. We believe that some capabilities are in fact innovative, since no other traffic generator offers them. Then, we describe the original architecture of D-ITG, which allows our traffic generator to achieve high performance. We hint at a comparison with other traffic generators and conclude with the ongoing work to add new features.
Several high-bandwidth network interfaces use Interrupt Coalescence (IC), i.e., they generate a single interrupt for multiple packets sent or received in a short time interval. IC decreases the per-packet interrupt processing overhead. However, IC also introduces queueing delays and alters the “dispersion” (i.e., interarrival time spacing) of packet pairs or trains. In this work, we first explain how IC works in two popular Gigabit Ethernet controllers. Then, we identify the negative effects of IC on active and passive network measurements. Specifically, we show that IC can affect active bandwidth estimation techniques, causing erroneous measurements. It can also alter the packet interarrivals in passive monitors that use commodity network interfaces. Based on the “signature” that IC leaves on the dispersion and one-way delays of packet trains, we show how to detect IC and how to filter its effects from raw measurements. Finally, we show that IC can be detrimental to TCP self-clocking, causing bursty delivery of ACKs and subsequent bursty transmission of data segments.
The Probe Gap Model (PGM) was proposed as a lightweight and fast available bandwidth estimation method. Measure- ment tools such as Delphi and Spruce are based on PGM. Compared to estimation methods that require multiple iter- ations with difierent probing rates, PGM uses a single prob- ing rate and it infers the available bandwidth from a direct relation between the input and output rates of measurement packet pairs. An important assumption behind the PGM model is that the measured path has a single bottleneck link that determines the available bandwidth of the end-to- end path. In this letter, we show that, even though PGM is accurate in the case of a single queue, it cannot estimate the available bandwidth of multi-hop paths, even if there is a single bottleneck in the path. Whether PGM is accurate or not depends on the routing of cross tra-c relative to the measurement tra-c. PGM is accurate when the cross tra-c follows the same path with the measurement tra-c. In the general case, however, PGM can signiflcantly underestimate the available bandwidth of an end-to-end path.
This paper presents a control-theoretic approach to reactive flow control in networks that do not reserve bandwidth. We assume a round-robin-like queue service discipline in the output queues of the network's switches, and propose deterministic and stochastic models for a single conversation in a network of such switches. These models motivate the Packet-Pair rate probing technique, and a provably stable rate-based flow control scheme. A Kalman state estimator is derived from discrete-time state space analysis, but there are difficulties in using the estimator in practice. These difficulties are overcome by a novel estimation scheme based on fuzzy logic. We then present a technique to extract and use additional information from the system to develop a continuous-time system model. This is used to design a variant of the control law that is also provably stable, and, in addition, takes control action as rapidly as possible. Finally, practical issues such as correcting parameter drift and coordination with window flow control are described.
Efficient and reliable available bandwidth measurement remains an important goal for many applications. In this paper we introduce an empirical bandwidth estimation tool based on neural networks. Training the neural network on simulation data, it provides reliable estimation of physical and available bandwidth for simulated single and multi-hop networks, in laboratory environment and among the real world conditions of the ETOMIC testbed.
In this paper, we propose a machine learning-based approach for estimating available bandwidth. We evaluate the approach via simulations using two probing models: a packet train probing model and a pathChirp-likeprobing model. The simulation results show that the former cannot yield accurate estimates in our system; however, using the pathChirp-like probing model, the proposed approach can estimate the available bandwidth with moderate traffic overhead more accurately than two widely used tools, pathChirp and Spruce. Moreover, we propose a normalization method that improves our approach's ability to estimate available bandwidth, even if there are no samples with similar properties to the measured path in the training dataset. The effectiveness and simplicity of this novel approach make it a promising scheme that goes a long way toward achieving accurate estimation of available bandwidth on Internet paths.
The packet pair technique estimates the capacity of a path
(bottleneck bandwidth) from the dispersion (spacing) experienced by two
back-to-back packets. We demonstrate that the dispersion of packet pairs
in loaded paths follows a multimodal distribution, and discuss the
queueing effects that cause the multiple modes. We show that the path
capacity is often not the global mode, and so it cannot be estimated
using standard statistical procedures. The effect of the size of the
probing packets is also investigated, showing that the conventional
wisdom of using maximum sized packet pairs is not optimal. We then study
the dispersion of long packet trains. Increasing the length of the
packet train reduces the measurement variance, but the estimates
converge to a value, referred to as the asymptotic dispersion rate
(ADR), that is lower than the capacity. We derive the effect of the
cross traffic in the dispersion of long packet trains, showing that the
ADR is not the available bandwidth in the path, as was assumed in
previous work. Putting all the pieces together, we present a capacity
estimation methodology that has been implemented in a tool called
pathrate
This paper analyzes the asymptotic behavior of packet-train probing over a multi-hop network path P carrying arbitrarily routed bursty cross-traffic flows. We examine the statistical mean of the packet-train output dispersions and its relationship to the input dispersion. We call this relationship the response curve of path P. We show that the real response curve Z is tightly lower-bounded by its multi-hop fluid counterpart F, obtained when every cross-traffic flow on P is hypothetically replaced with a constant-rate fluid flow of the same average intensity and routing pattern. The real curve Z asymptotically approaches its fluid counterpart F as probing packet size or packet train length increases. Most existing measurement techniques are based upon the single-hop fluid curve S associated with the bottleneck link in P. We note that the curve S coincides with F in a certain large-dispersion input range, but falls below F in the remaining small-dispersion input ranges. As an implication of these findings, we show that bursty cross-traffic in multi-hop paths causes negative bias (asymptotic underestimation) to most existing techniques. This bias can be mitigated by reducing the deviation of Z from S using large packet size or long packet-trains. However, the bias is not completely removable for the techniques that use the portion of S that falls below F.
Most existing available-bandwidth measurement techniques are justified using a constant-rate fluid cross-traffic model. To achieve a better understanding of the performance of current bandwidth measurement techniques in general traffic conditions, this paper presents a queueing-theoretic foundation of single-hop packet-train bandwidth estimation under bursty arrivals of discrete cross-traffic packets. We analyze the statistical mean of the packet-train output dispersion and its mathematical relationship to the input dispersion, which we call the probing-response curve. This analysis allows us to prove that the single-hop response curve in bursty cross-traffic deviates from that obtained under fluid cross traffic of the same average intensity and to demonstrate that this may lead to significant measurement bias in certain estimation techniques based on fluid models. We conclude the paper by showing, both analytically and experimentally, that the response-curve deviation vanishes as the packet-train length or probing packet size increases, where the vanishing rate is decided by the burstiness of cross-traffic.
The available bandwidth (avail-bw) in a network path is of major importance in congestion control, streaming applications, quality-of-service verification, server selection, and overlay networks. We describe an end-to-end methodology, called self-loading periodic streams (SLoPS), for measuring avail-bw. The basic idea in SLoPS is that the one-way delays of a periodic packet stream show an increasing trend when the stream's rate is higher than the avail-bw. We have implemented SLoPS in a tool called pathload. The accuracy of the tool has been evaluated with both simulations and experiments over real-world Internet paths. Pathload is nonintrusive, meaning that it does not cause significant increases in the network utilization, delays, or losses. We used pathload to evaluate the variability ("dynamics") of the avail-bw in Internet paths. The avail-bw becomes significantly more variable in heavily utilized paths, as well as in paths with limited capacity (probably due to a lower degree of statistical multiplexing). We finally examine the relation between avail-bw and TCP throughput. A persistent TCP connection can be used to measure roughly the avail-bw in a path, but TCP saturates the path and increases significantly the path delays and jitter.
The packet pair mechanism has been shown to be a reliable method to measure the bottleneck link capacity on a network path, but its use for measuring available bandwidth is more challenging. In this paper, we use modeling, measurements, and simulations to better characterize the interaction between probing packets and the competing network traffic. We first construct a simple model to understand how competing traffic changes the probing packet gap for a single-hop network. The gap model shows that the initial probing gap is a critical parameter when using packet pairs to estimate available bandwidth. Based on this insight, we present two available bandwidth measurement techniques, the initial gap increasing (IGI) method and the packet transmission rate (PTR) method. We use extensive Internet measurements to show that these techniques estimate available bandwidth faster than existing techniques such as Pathload, with comparable accuracy. Finally, using both Internet measurements and ns simulations, we explore how the measurement accuracy of active probing is affected by factors such as the probing packet size, the length of probing packet train, and the competing traffic on links other than the tight link.
Tools for estimating end-to-end available bandwidth (AB) send out a train of packets and observe how inter-packet gaps change over a given network path. In ultra-high speed networks, the fine inter-packet gaps are fairly susceptible to noise introduced by transient queuing and bursty cross-traffic. Past work uses smoothing heuristics to alleviate the impact of noise, but at the cost of requiring large packet trains. In this paper, we consider a machine-learning approach for learning the AB from noisy inter-packet gaps. We conduct extensive experimental evaluations on a 10 Gbps testbed, and find that supervised learning can help realize ultra-high speed bandwidth estimation with more accuracy and smaller packet trains than the state of the art. Further, we find that when training is based on: (i) more bursty cross-traffic, (ii) extreme configurations of interrupt coalescence, a machine learning framework is fairly robust to the cross-traffic, NIC platform, and configuration of NIC parameters.
The available bandwidth (avail-bw) in a network path is of major importance in congestion control, streaming applications, QoS verification, server selection, and overlay networks. We describe an end-to-end methodology, called Self-Loading Periodic Streams (SLoPS), for measuring avail-bw. The basic idea in SLoPS is that the one-way delays of a periodic packet stream show an increasing trend when the stream's rate is higher than the avail-bw. We implemented SLoPS in a tool called pathload. The accuracy of the tool has been evaluated with both simulations and experiments over real-world Internet paths. Pathload is non-intrusive, meaning that it does not cause significant increases in the network utilization, delays, or losses. We used pathload to evaluate the variability ('dynamics') of the avail-bw in some paths that cross USA and Europe. The avail-bw becomes significantly more variable in heavily utilized paths, as well as in paths with limited capacity (probably due to a lower degree of statistical multiplexing). We finally examine the relation between avail-bw and TCP throughput. A persistent TCP connection can be used to roughly measure the avail-bw in a path, but TCP saturates the path, and increases significantly the path delays and jitter.
Numerous methods for available bandwidth estimation have been developed for wireline networks, and their effectiveness is well-documented. However, most methods fail to predict bandwidth availability reliably in a wireless setting. It is accepted that the increased variability of wireless channel conditions makes bandwidth estimation more difficult. However, a (satisfactory) explanation why these methods are failing is missing. This paper seeks to provide insights into the problem of bandwidth estimation in wireless networks or, more broadly, in networks with random service. We express bandwidth availability in terms of bounding functions with a defined violation probability. Exploiting properties of a stochastic min-plus linear system theory, the task of bandwidth estimation is formulated as inferring an unknown bounding function from measurements of probing traffic. We present derivations showing that simply using the expected value of the available bandwidth in networks with random service leads to a systematic overestimation of the traffic departures. Furthermore, we show that in a multihop setting with random service at each node, available bandwidth estimates requires observations over (in principle infinitely) long time periods. We propose a new estimation method for random service that is based on iterative constant-rate probes that take advantage of statistical methods. We show how our estimation method can be realized to achieve both good accuracy and confidence levels. We evaluate our method for wired single-and multihop networks, as well as for wireless networks.
This paper describes the active available bandwidth measurement tool DietTopp. It measures the available bandwidth and the link capacity on an end-to- end path having one bottleneck link. DietTopp is based on a simplified TOPP method. This paper describe and motivate the simplifications and assumptions made to TOPP. Further, the paper describes some of the DietTopp implementation issues. A first evaluation of DietTopp in a testbed scenario is made. Within this evaluation the performance and measurement accuracy of DietTopp is compared to the state-of-the-art tools Pathload and Pathrate. We show that DietTopp gives fast and accurate estima- tions of both the available bandwidth and the link capacity of the bottleneck link.
Emulab is a large-scale, remotely-accessible network and distributed systems testbed used by over a thousand researchers around the world. In Emulab, users create "experiments" composed of arbitrarily interconnected groups of dedicated machines that are automatically configured according to user specifications. In the last year alone, users have run over 18,000 such experiments, expecting consistent and correct behavior in the face of the ever-evolving 500,000 line code base and 3,000 discrete hardware components that comprise Emulab. We have found normal testing to be insufficient to meet these expectations and have therefore provided continuous, automatic validation. This paper describes Linktest, an integral part of our validation framework that is responsible for end-to-end validation during the configuration of every experiment. Developing and deploying such a validation approach faces numerous challenges, including the need for a code path entirely independent of the rest of the Emulab software. We describe our system's motivation, its design and implementation and our experience.
In this paper we address the problem of online bandwidth estimation in wired and wireless LANs. To this end, we employ active
probing, i.e. we continuously inject packet probes into the network. We present the key challenges and analyze the trade-offs
between fast change detection and estimate smoothness. We show the benefit of using Kalman filtering to obtain optimal estimates
under certain conditions and provide a procedure for parameterizing the filter with respect to specific use cases. Furthermore,
we evaluate the influence of probing train length on the results. Based on our findings we developed a tool implementing the
presented methodology. We support our theoretical results by a number of real-world measurements as well as simulations.
This paper presents a new method, BART (bandwidth available in real-time), for estimating the end-to-end available bandwidth over a network path. It estimates bandwidth quasi-continuously, in real-time. The method has also been implemented as a tool. It relies on self-induced congestion, and repeatedly samples the available bandwidth of the network path with sequences of probe packet pairs, sent at randomized rates. BART requires little computation in each iteration, is lightweight with respect to memory requirements, and adds only a small amount of probe traffic. The BART method uses Kalman filtering, which enables real-time estimation (a.k.a. tracking). It maintains a current estimate, which is incrementally improved with each new measurement of the inter-packet time separations in a sequence of probe packet pairs. The measurement model has a strong non-linearity, and would not at first sight be considered suitable for Kalman filtering, but we show how this non-linearity can be handled. BART may be tuned according to the specific needs of the measurement application, such as agility vs. stability of the estimate. We have tested an implementation of BART in a physical test network with carefully controlled cross traffic, with good accuracy and agreement. Test measurements have also been performed over the Internet. We compare the performance of BART with that of pathChirp, a state-of-the-art tool for measuring end-to-end available bandwidth in real-time
This paper presents a new foundational approach to reason about available bandwidth estimation as the analysis of a min-plus linear system. The available bandwidth of a link or complete path is expressed in terms of a service curve , which is a function that appears in the network calculus to express the service available to a traffic flow. The service curve is estimated based on measurements of a sequence of probing packets or passive measurements of a sample path of arrivals. It is shown that existing bandwidth estimation methods can be derived in the min-plus algebra of the network calculus, thus providing further mathematical justification for these methods. Principal difficulties of estimating available bandwidth from measurements of network probes are related to potential nonlinearities of the underlying network. When networks are viewed as systems that operate either in a linear or in a nonlinear regime, it is argued that probing schemes extract the most information at a point when the network crosses from a linear to a nonlinear regime. Experiments on the Emulab testbed at the University of Utah, Salt Lake City, evaluate the robustness of the system-theoretic interpretation of networks in practice. Multinode experiments evaluate how well the convolution operation of the min-plus algebra provides estimates for the available bandwidth of a path from estimates of individual links.
Most recent developments of computer sciences, such as web services, Grid, peer-to-peer, and mobile computing, are network-based computing. Their applicability depends on the availability of the underlying network bandwidth. However, network resources are shared and the available network bandwidth varies with time. There is no satisfactory solution available for network performance predictions. This lack of prediction limits the applicability of network-based computing, especially for Grid computing where concurrent remote processing is essential. In this study, we propose an Artificial Neural Network (ANN) based approach for network performance prediction. The ANN mechanism has been tested on classical trace files and compared with the well-known system NWS (Network Weather Service) for performance. Experimental results show the ANN approach always provides an improved prediction over that of NWS. ANN has a real potential in network computing.
We present a network friendly bandwidth measurement method, TOPP,
that is based on active probing and includes analysis by segmented
regression. This method can estimate two complementing available
bandwidth metrics in addition to the link bandwidth of the congested
link. Contrary to traditional packet pair estimates of the bottleneck
link bandwidth, our estimate is not limited by the rate at which we can
inject probe packets into the network. We also show that our method is
able to detect bottlenecks that are invisible to methods such as the
C-probe. Further more, we describe scenarios where our analysis method
is able to calculate bandwidth estimates for several congested hops
based on a single end-to-end probe session
We discuss findings from a large-scale study of Internet packet dynamics conducted by tracing 20000 TCP bulk transfers between 35 Internet sites. Because we traced each 100-kbyte transfer at both the sender and the receiver, the measurements allow us to distinguish between the end-to-end behavior due to the different directions of the Internet paths, which often exhibit asymmetries. We: (1) characterize the prevalence of unusual network events such as out-of-order delivery and packet replication; (2) discuss a robust receiver-based algorithm for estimating “bottleneck bandwidth” that addresses deficiencies discovered in techniques based on “packet pair;” (3) investigate patterns of packet loss, finding that loss events are not well modeled as independent and, furthermore, that the distribution of the duration of loss events exhibits infinite variance; and (4) analyze variations in packet transit delays as indicators of congestion periods, finding that congestion periods also span a wide range of time scales
A measurement study of available bandwidth estimation tools
- J Strauss
- D Katabi
- F Kaashoek