Figure 2 - uploaded by Roland Ewald
Content may be subject to copyright.
Example decision tree representing the function k . It is traversed from top to bottom until a leaf is reached, and works on both problem and state features. At first, the (relative) range is considered: if it is below or equal to 0 : 6 , the left sub-tree is selected, otherwise the right one. In the left sub-tree, a decision is made depending on the state feature of estimator a 1 , by determining whether it could estimate a steady state mean, stored in its state, i.e., s 1 : res = \ . In the right sub-tree, the standard deviation is requested. If the deviation falls below 0 : 1 , the state feature of 

Example decision tree representing the function k . It is traversed from top to bottom until a leaf is reached, and works on both problem and state features. At first, the (relative) range is considered: if it is below or equal to 0 : 6 , the left sub-tree is selected, otherwise the right one. In the left sub-tree, a decision is made depending on the state feature of estimator a 1 , by determining whether it could estimate a steady state mean, stored in its state, i.e., s 1 : res = \ . In the right sub-tree, the standard deviation is requested. If the deviation falls below 0 : 1 , the state feature of 

Source publication
Article
Full-text available
Simulation experiments involve various sub-tasks, e.g., parameter optimization, simulation execution, or output data analysis. Many algorithms can be applied to such tasks, but their performance depends on the given problem. Steady state estimation in systems biology is a typical example for this: several estimators have been proposed, each with it...

Contexts in source publication

Context 1
... in Section ‘Experiments: A Synthetic Steady State Estinator’). F P contains two problem feature extractors, one for the standard deviation and one for the range, i.e., the distance between minimum and maximum value, of the time series. F S contains a state feature extractor that retrieves the estimated steady state mean from the algorithms in A , stored in their states, which may be \ to denote that none was found yet. The selection function s is trivial, it always selects both algorithms: s ( H F ) ~ f a 1 , a 2 g . The composition function k is based on a decision tree (e.g., [47]), which is generated from previously collected performance data, where a 1 and a 2 have been applied to representative problems. Figure 2 shows a hypothetical decision tree that could result from such a training process. 2.3.1 Plugin -based Realization. The process of SPS creation, deployment, and usage should be automated as much as possible. Our prototype is based on the open-source modeling and simulation framework James II [15,48] (See . org). We chose James II as base for the implementation of our concept, as its plugin structure allows a separated and flexible application of algorithms. Furthermore, James II is based on Java, a very popular, platform-independent programming language including a static type-system that is beneficial for the treatment of different algorithm types by maintaining well-defined interfaces. However, our concept can be realized for any simulation system or programming language providing plugins that can be analyzed and applied by reasoning mechanisms. In James II, algorithms and data structures that are encapsu- lated in plugins are managed by a central registry. The registry distinguishes plugins based on their plugin type , which corresponds to the functionality they offer. Broadly speaking, plugins of a certain plugin type provide alternative implementations of a particular Java interface. James II allows to add new plugins and new plugin types. Both are declared in XML files, loaded during start-up, and can be discovered at runtime, by querying the central registry. This makes it easy for developers to re-use and extend the functionality of the framework. The prototype of our synthetic problem solver SPS ~ S A , F P , F S , s , k T (see Equation 3) is based on the following plugins and plugin types: 1. A : a list of problem solver algorithms, i.e., plugins of type problem solver . The corresponding Java interface declares the solve function (see Equation 1). 2. F P , F S : lists of problem feature extractors and state feature extractors, i.e., plugins of type feature extractor . The distinction between problem and state feature extractors is maintained by defining the type of object to which an extractor can be applied. 3. s , k : selection and composition function are implemented by plugins of type result composer . The two functions are often related (see Section ‘‘Mapping Synthetic Problem Solvers to Existing Composition Schemes’’), so we combine them to a single component that fully determines the composition. A , P , and S contain predefined plugins, but the composition logic depends on the characteristics of the problems to be solved (see Section 2.1.1). Hence, users should be able to (re-)generate s and k automatically. This is described in the following. Functions. The functions s and k of an SPS instance can realize various kinds of composition (see Section ‘‘Mapping Synthetic Problem Solvers to Existing Composition Schemes’’). They are typically generated from empirical data, by considering the performance of base-line solvers on previously encountered problems. We assume that a set of such training problems is available. Figure 3 shows an overview of the automatic generation process. At first, the base-line problem solvers in A are applied individually to the given training problems. Both problem features and state features are recorded during this evaluation. State features can also include performance measurements, such as memory consumption or execution time. The features are extracted automatically and written to a dedicated performance database [49], which is integrated into the experimentation layer of James II [16]. This makes it easy to conduct large-scale performance evaluations with various experiment designs, so that even large sets of training problems can be analyzed conveniently. Often, an additional data aggregation step is required. For example, generating a k function for an algorithm ensemble requires to aggregate the data tuples of all individual problem solvers (see Section 2.4.3). Aggregation may also be necessary if either algorithm performances or training problems are stochastic. Given the aggregated data, we can now generate different representations of the s and k functions. This ‘Data Analysis’ step (see Figure 3) is handled by plugins. For example, one plugin could implement a machine learning algorithm to create decision trees like the one shown in Figure 2. Several machine learning algorithms are already available as plugins, e.g., implementations from the WEKA toolkit [50]. Then, the representations of s and k are evaluated on previously unseen problems. This step ensures, for example, that a learning method is indeed suitable to the domain at hand, e.g., by estimating its prediction error on previously unseen problems. Like the data analysis step, this procedure is fully customizable via plugins. In a final step, the best-performing representations of s and k are incorporated in a new SPS instance. We describe its deployment to James II in the next section. 2.3.3 Deployment to the James II Registry. The result of the process depicted in Figure 3 is an instance of the synthetic problem solver, which needs to be deployed as a new plugin to the James II registry. However, the way in which James II plugins are created and loaded is rather static. Typically, plugins are declared in XML files that contain relevant Java class names and some predefined meta-data, e.g., regarding parameters [15]. An SPS instance, however, is not defined by a class, but instead by specific instances of its base-line problem solvers ( A ), its feature extractors ( F P and F S ), and its selection and composition functions ( s and k ). Therefore, we extend the registry to support a more flexible definition of plugins. SPS instances are wrapped in so-called synthetic plugins , which are handled by an additional management component. Synthetic plugins are stored to a local file, which is read during start-up. While they are kept in a dedicated data structure, i.e., separated from the other James II plugins, queries to discover a plugin of a certain type now also consider all synthetic plugins of that type. This process is transparent to application code, i.e., an SPS instance can be applied throughout James II without code changes. After an SPS has been trained and deployed (typically, by an expert for the problem domain or the algorithms at hand), users do not have to care about selecting the best suited algorithm for a problem, or how to apply trained compositions. Instead, they simply use the deployed synthetic plugin, which returns an SPS instance comprising the composition. Synthetic plugin and SPS instance are thus used similarly to the standard plugins and algorithms offered by James II. Synthetic plugins may provide additional meta-data that is important for selecting between them. In case of steady state estimation, this could include the statistical properties of the time series used for training. We now elaborate how the SPS can be configured to realize the composition approaches discussed in Section 2.1.1. The simplicity of these mappings makes us confident that this approach could be useful for many other problems encountered in computational systems biology. 2.4.1 Algorithm Selection Problem. The first element of an SPS ~ S A , F P , F S , s , k T , A , corresponds to the set of algorithms A in the algorithm selection problem (see Section 2.1.1). However, each algorithm in A has to comply with the problem solver interface (see Equation 1). The problem feature extractors in F P are analogous to feature extractors in the ASP, but F P would also include a problem feature extractor that returns user criteria. The set of state feature extractors, F S , contains a single extractor that retrieves the overall results from the state of the selected base-line problem solver. As only one base-line problem solver shall be selected and used, the selection function s picks a single element from A and thus realizes a selection mapping for the ASP. Typically, s is generated by analyzing algorithm performance on training problems. This analysis relies, at least implicitly, on some measure of performance, which is explicitly considered by the ASP but has no direct counterpart in the SPS. The composition function k returns the results extracted from the state of the selected base-line problem solver. As a sample setup, consider a set A of stochastic simulation algorithms (SSAs) for chemical reaction networks [51]. SSA performance strongly depends on the given model, so that automatically selecting a suitable algorithm can considerably improve execution time [12]. The problem space P it contains all chemical reaction networks that can be simulated with SSAs. Problem feature extractors in F P may retrieve the number of distinct species and reactions from a model, or a measure of its stiffness. An additional user criterion could, for example, account for approximative SSA variants and specify an acceptable trade-off between execution speed and accuracy. The state feature extractor in F S would retrieve the generated simulation trajectory from an SSA state, and the selection function s could be generated via supervised learning on previously recorded execution times. A similar approach was pursued in [52] for spatial SSAs. The composition function k would return the generated ...
Context 2
... has been detected [6]. 5. Running Mean: assumes the end of the warm-up phase as soon as the change in the running mean falls below a given threshold [58]. 6. Batch Mean: divides a time series into batches and the batches into two groups. Assumes the end of the warm-up phase as soon as the distributions of the variances for the two groups are close enough [9]. 7. Crossing Mean: counts the crosses of a time series and its running mean. As soon as a given amount of crosses occurred, the end of the warm-up phase is assumed [7]. 8. Stop Crossing Mean: counts the crosses of time series and running mean. As soon as no crosses happened for a given length, the end of the warm-up phase is assumed. 9. Schruben’s Test: estimates the stationarity of a time series to identify its warm-up phase [8]. 10.Moving Windows: moves a window through a time series and updates the mean according to the values inside the window. If the standard deviation falls below a given threshold, the warm- up phase has been detected [59]. Methods 1–3 have been proposed by [57] for automated experimentation, as they do not require a careful configuration. The others have been proposed during the last four decades and are still in use, which illustrates the difficulty of selecting a suitable steady state estimator for a given problem. We realized all methods as plugins of James II, and use them with their default parameters. The result of each problem solver is the estimated steady state mean, or \ if no steady state mean could be estimated. Hence, RES ~ R | f \ g . The request is a boolean value denoting whether additional data points are required for estimation, hence REQ ~ f true , false g . 3.1.2 Problem Feature Extractors ( F P ). The feature extractors shall characterize two key aspects of steady state mean estimation on a time series : the amount of noise and the shape of the initialization bias. We implemented the following time series feature extractors in James II: 1. Count of values above sample mean, to measure the amount of values increasing the positive bias. 2. Count of values below sample mean, to measure the amount of values increasing the negative bias. 3. Count of values below sample mean subtracted from those above sample mean, to measure whether positive or negative values dominate the time series. 4. Maximum positive distance to sample mean, to measure the impact of values increasing the positive bias. 5. Minimum negative distance to sample mean, to measure the impact of values increasing the negative bias. 6. Maximum absolute distance to sample mean, to measure the overall bias. 7. Portmanteau Test [60], to get a measure for the autocorrela- tion, and therefore an indication whether patterns in the curve are repeated. Features 1–6 characterize the type of bias (mainly positive or negative) in the time series, feature 7 gives an idea about the noise. We normalized the features in order to scale with different time series sizes. Features 1–3 are divided by the size of the time series and features 4–7 by its value range. For a better characterization of the initialization bias in the presence of noise, we implemented additional extractors that calculate the above features from a smoothed trend curve of the time series. We used an exponential smoothing algorithm [61] for creating the trend curve. Previous experiments showed that feature 6 extracted from the time series (not on the trend curve) biased the learning process of the composition function (see Section 3.1.5), leading to overfitted results. Hence, we leave this particular feature extractor out in our experiments, so that we extract 6 z 7 ~ 13 problem features overall. 3.1.3 State Feature Extractors ( F S ). Regarding state features, we are particularly interested in the detection result of a steady state estimator, i.e., whether it found the end of the warm- up phase in a time series or not. This detection result is retrieved by a state feature extractor f d . The extractor f d returns 1 if the request element of the given state equals false, i.e., the end of the warm-up phase has been detected, so that a steady state can now be estimated and no further data is required. Otherwise, f d returns 0 if more data is requested, i.e., the end of the warm-up phase has not been detected yet: 3.1.4 Selection Function ( s ). As no selection of base-line solvers is considered, the selection function returns all given baseline solvers: 3.1.5 Composition Function ( k ). We generate a composition function with a wrapper for WEKA’s J48, which is an implementation of the C4.5 decision tree algorithm [62]. The basic idea behind decision trees is to identify those attributes (i.e., features) of training set instances that contain most information regarding the class attribute (i.e., end of warm-up phase detected or not). This attribute selection is done recursively, to construct a tree of nodes representing decisions on attributes. However, the used tree differs from the example tree of Figure 2, as it decides on the end of the problem time series ’ warm-up phase, instead of deciding on the steady state estimator that shall be used for creating estimation results. This approach offers further opportu- nities for composition, as the results of different steady state estimators can be combined after the end of the warm-up phase has been found. In the present example, they have been combined by using an equally weighted portfolio of the base-line problem solvers. To generate training data for the J48, we use a problem generator that creates sample problems. It follows the evaluation approaches of [9], [63], [64], and [57] that generate time series over several successive steps, instead of using an M/M/1 queue as, e.g., [65], [66], and [67] do. The reason for this decision is its straightforward parametrization, which allows for a direct control of several key features (see below). The problem generator has the following structure: It generates time series with bias of length l b , trend t , and shape s . The noise of the time series is induced by a random number generator and varies between { n and z n . The series is of size l o and the numbers are cross correlated with factor c . We selected trend and shape to cover the most relevant types of time series, orienting our selection towards previous studies [9,57,63,64]. We pursued a more fine-grained investigation of bias during the warm-up phase, by distinguishing between two dimensions. The first dimension is bias trend t which includes two variants: constant and quadratic. The second dimension is bias shape s , where three variants are considered: a straight line shape, an oscillating shape, and a random shape. By combining the shapes and trends, six types of bias can be created. For instance, a quadratic trend with an oscillating shape means that during the warm-up phase, the time series initial bias oscillates with a quadratically decreasing amplitude, as shown in Figure 4. In addition to bias, three other factors have been varied in our experiments: noise, cross-correlation, and bias length. We tested bias lengths from 0 to 150 percent of the time series length. Bias length of more than 100 percent allows to investigate the ability of an estimator to handle heavily biased time series. Furthermore, as stochasticity plays a role in steady state estimation, we tested different noise levels to investigate the robustness of the estimators against them. The noise levels range from 0 to 10 percent, where, e.g., 10 percent noise means that a random value between { 10 and 10 percent is added to each value of the time series. The noise has been generated with an auto-correlation factor of 0 : 5 to get more realistic random numbers that are not independent and identically distributed. Finally the length of the time series has been varied between 1,000 and 10,000 time points. We generated 8,640 problem definitions, each being used to randomly generate 10 time series to tackle stochastic bias. Hence, each of the used steady state estimators has been applied to 86,400 time series. 3.1.6 Problem Solver Evaluation. For simplicity, the steady state estimators used in this study are considered as monolithic entities, i.e., each tested steady state estimator is given with a fixed (default) parametrization. Hence, a distinct consideration of the influence of different parameter settings is not done, but should be the subject of follow-up studies. We compare the performance of the created SPS instance to those of the base-line problem solvers, by applying them to real simulation data. To do so, we execute five p -Calculus models [68] that produce time series as input for the steady state estimators. The models are examples models for the Stochastic Pi Machine (SPiM) ...
Context 3
... A composition function k : H F ? RES | REQ composes the overall results of the selected base-line solvers and generates further requests. For steady state estimation, a selection function s could select all estimators that have not yet detected the end of the warm-up phase. Likewise, a composition function k could request further data points until the majority of estimators has detected an end of the warm-up phase, and then average their estimates to calculate the overall result. The actual state space ( AS ) of the synthetic problem solver consists of the feature history and the base-line solver states, so that its overall state S synth is defined as Each base-line solver could have a similar state, i.e., synthetic problem solvers can be nested. 2.2.2 Implementation. We can now implement the solve function of the generic interface (see Equation 1) for a synthetic problem solver SPS ~ S A , F P , F S , s , k T . Algorithm 1 in Table 1 shows the pseudo-code. It starts with extracting the set E P of problem features (see line 6, Algorithm 1 in Table 1). Then, the function s selects suitable problem solvers, based on the extracted problem features E P and the feature history H f (line 9). Afterwards, the solve method of the selected base-line solvers is applied to retrieve their successor states (line 12), while the states of the other base-line solvers remain unchanged (line 13). State features are extracted from the successor states, resulting in the set E S of extracted state features (line 16). Problem and state features are appended to the feature history H f , leading to a successor feature history H f ’ (line 19) that is used for composing the next result and request of the SPS by applying the composition function k (line 22). Finally, the successor state s ’ is created by combining feature history, base-line solver states, result, and request (line 25). Figure 1 shows the data flows during an execution of solve. 2.2.3 An Example SPS for Steady State Estimation. As mentioned before, simulation-based steady state estimation works on time series. In each iteration, an additional part of a time series is created by simulation and investigated by the estimator, i.e., P it ~ R n . The result is an estimate of some steady state statistic, in our case the steady state mean. Therefore, RES ~ R | f \ g and \ again denotes that no steady state mean could be estimated. The request is a boolean value denoting whether additional data are required for estimation, hence REQ ~ f true , false g . Our sample SPS ~ S A , F P , F S , s , k T uses two base-line problem solvers, i.e., A ~ f a 1 , a 2 g , where a 1 is the MSER steady state estimator [10] and a 2 is Schruben’s test steady state estimator [8] (this is a very brief example; we present a more realistic case study in Section ‘Experiments: A Synthetic Steady State Estinator’). F P contains two problem feature extractors, one for the standard deviation and one for the range, i.e., the distance between minimum and maximum value, of the time series. F S contains a state feature extractor that retrieves the estimated steady state mean from the algorithms in A , stored in their states, which may be \ to denote that none was found yet. The selection function s is trivial, it always selects both algorithms: s ( H F ) ~ f a 1 , a 2 g . The composition function k is based on a decision tree (e.g., [47]), which is generated from previously collected performance data, where a 1 and a 2 have been applied to representative problems. Figure 2 shows a hypothetical decision tree that could result from such a training process. 2.3.1 Plugin -based Realization. The process of SPS creation, deployment, and usage should be automated as much as possible. Our prototype is based on the open-source modeling and simulation framework James II [15,48] (See . org). We chose James II as base for the implementation of our concept, as its plugin structure allows a separated and flexible application of algorithms. Furthermore, James II is based on Java, a very popular, platform-independent programming language including a static type-system that is beneficial for the treatment of different algorithm types by maintaining well-defined interfaces. However, our concept can be realized for any simulation system or programming language providing plugins that can be analyzed and applied by reasoning mechanisms. In James II, algorithms and data structures that are encapsu- lated in plugins are managed by a central registry. The registry distinguishes plugins based on their plugin type , which corresponds to the functionality they offer. Broadly speaking, plugins of a certain plugin type provide alternative implementations of a particular Java interface. James II allows to add new plugins and new plugin types. Both are declared in XML files, loaded during start-up, and can be discovered at runtime, by querying the central registry. This makes it easy for developers to re-use and extend the functionality of the framework. The prototype of our synthetic problem solver SPS ~ S A , F P , F S , s , k T (see Equation 3) is based on the following plugins and plugin types: 1. A : a list of problem solver algorithms, i.e., plugins of type problem solver . The corresponding Java interface declares the solve function (see Equation 1). 2. F P , F S : lists of problem feature extractors and state feature extractors, i.e., plugins of type feature extractor . The distinction between problem and state feature extractors is maintained by defining the type of object to which an extractor can be applied. 3. s , k : selection and composition function are implemented by plugins of type result composer . The two functions are often related (see Section ‘‘Mapping Synthetic Problem Solvers to Existing Composition Schemes’’), so we combine them to a single component that fully determines the composition. A , P , and S contain predefined plugins, but the composition logic depends on the characteristics of the problems to be solved (see Section 2.1.1). Hence, users should be able to (re-)generate s and k automatically. This is described in the following. Functions. The functions s and k of an SPS instance can realize various kinds of composition (see Section ‘‘Mapping Synthetic Problem Solvers to Existing Composition Schemes’’). They are typically generated from empirical data, by considering the performance of base-line solvers on previously encountered problems. We assume that a set of such training problems is available. Figure 3 shows an overview of the automatic generation process. At first, the base-line problem solvers in A are applied individually to the given training problems. Both problem features and state features are recorded during this evaluation. State features can also include performance measurements, such as memory consumption or execution time. The features are extracted automatically and written to a dedicated performance database [49], which is integrated into the experimentation layer of James II [16]. This makes it easy to conduct large-scale performance evaluations with various experiment designs, so that even large sets of training problems can be analyzed conveniently. Often, an additional data aggregation step is required. For example, generating a k function for an algorithm ensemble requires to aggregate the data tuples of all individual problem solvers (see Section 2.4.3). Aggregation may also be necessary if either algorithm performances or training problems are stochastic. Given the aggregated data, we can now generate different representations of the s and k functions. This ‘Data Analysis’ step (see Figure 3) is handled by plugins. For example, one plugin could implement a machine learning algorithm to create decision trees like the one shown in Figure 2. Several machine learning algorithms are already available as plugins, e.g., implementations from the WEKA toolkit [50]. Then, the representations of s and k are evaluated on previously unseen problems. This step ensures, for example, that a learning method is indeed suitable to the domain at hand, e.g., by estimating its prediction error on previously unseen problems. Like the data analysis step, this procedure is fully customizable via plugins. In a final step, the best-performing representations of s and k are incorporated in a new SPS instance. We describe its deployment to James II in the next section. 2.3.3 Deployment to the James II Registry. The result of the process depicted in Figure 3 is an ...

Similar publications

Conference Paper
Full-text available
This paper focuses on Distributed State Estimation over a peer-to-peer sensor network composed by possible low-computational sensors. We propose a new l-step Neighbourhood Distributed Moving Horizon Estimation technique with fused arrival cost and pre-estimation, improving the accuracy of the estimation, while reducing the computation time compared...

Citations

... The automatic selection of methods in modeling and simulation has been investigated with the goal of achieving more robust results or faster runtimes. E.g., in the case of steady state estimation, algorithm portfolios have been used (Leye et al. 2014). The idea of algorithm portfolios was to apply a set of algorithms to the same problem and to combine their results. ...
... A steady-state (SS) exists when the variables of a system are "constant" across time, with their rates of change having slopes of approximately zero. Steady state detection (SSD) is important in diverse fields: chemical engineering, 1 materials science, 2 biology, 3 queuing management, 4 fault detection, 5,6 econometrics, 7,8 medical applications, 9 and other fields 10,11 . A number of methods exist for SSD [11][12][13] and the related concept of convergence. ...
Article
Steady‐state detection is of vital importance for numerous simulations in chemical engineering, as well as also other fields of science, engineering, and finance – particularly when the full timescale of interest cannot be measured or simulated. We present a breakthrough for estimating the number of data points required before successful steady‐state detection is feasible. Using an initial window of data, the method enables predicting the prerequisites for steady state detection (ppSSD), given as a number of data points. The method is shown to be accurate for data with realistic distributions (uniform, normal, and sine‐wave), and data from actual kinetic Monte Carlo simulations. Users need only to use the algebraic equations derived and provided in this work to estimate the required number of data points for relevant steady‐state detection. This article is protected by copyright. All rights reserved.
Article
Simulation has become, in many application areas, a sine-qua-non. Most recently, COVID-19 has underlined the importance of simulation studies and limitations in current practices and methods. We identify four goals of methodological work for addressing these limitations. The first is to provide better support for capturing, representing, and evaluating the context of simulation studies, including research questions, assumptions, requirements, and activities contributing to a simulation study. In addition, the composition of simulation models and other simulation studies’ products must be supported beyond syntactical coherence, including aspects of semantics and purpose, enabling their effective reuse. A higher degree of automating simulation studies will contribute to more systematic, standardized simulation studies and their efficiency. Finally, it is essential to invest increased effort into effectively communicating results and the processes involved in simulation studies to enable their use in research and decision-making. These goals are not pursued independently of each other, but they will benefit from and sometimes even rely on advances in other subfields. In the present paper, we explore the basis and interdependencies evident in current research and practice and delineate future research directions based on these considerations.