Page 1

Sloppy Models, Parameter Uncertainty, and the Role of

Experimental Design

Joshua F. Apgar1,2,†, David K. Witmer2,3,†, Forest M. White1,4, and Bruce Tidor1,2,3,*

1Department of Biological Engineering

2Computer Science and Artificial Intelligence Laboratory

3Department of Electrical Engineering and Computer Science

4David H. Koch Institute for Integrative Cancer Research, Massachusetts Institute of Technology,

Cambridge, Massachusetts 02139, USA

Summary

Computational models are increasingly used to understand and predict complex biological

phenomena. These models contain many unknown parameters, at least some of which are difficult

to measure directly, and instead are estimated by fitting to time-course data. Previous work has

suggested that even with precise data sets, many parameters are unknowable by trajectory

measurements. We examined this question in the context of a pathway model of epidermal growth

factor (EGF) and neuronal growth factor (NGF) signaling. Computationally, we examined a

palette of experimental perturbations that included different doses of EGF and NGF as well as

single and multiple gene knockdowns and overexpressions. While no single experiment could

accurately estimate all of the parameters, experimental design methodology identified a set of five

complementary experiments that could. These results suggest optimism for the prospects for

calibrating even large models, that the success of parameter estimation is intimately linked to the

experimental perturbations used, and that experimental design methodology is important for

parameter fitting of biological models and likely for the accuracy that can be expected for them.

Keywords

epidermal growth factor; model calibration; neuronal growth factor; parameter estimation; sloppy

models

Introduction

One of the goals of systems biology is the construction of computational models that can

accurately predict the response of a biological system to novel stimuli.1-3 Such models serve

to encapsulate our current understanding of biological systems, can indicate gaps in that

understanding, and have the potential to provide a basis for the rational design of

experiments,4,5 clinical interventions,6,7 and synthetic biological systems.8 There are many

varieties of computational models ranging from abstracted data-driven models to highly

detailed molecular-mechanics ones. In this report we focus on the popular class of ordinary

differential equation (ODE) models9-16 typically used to describe systems at the biochemical

and pharmacokinetic level but which are also appropriate at more abstract levels.

*Corresponding Author: Bruce Tidor, MIT Room 32-212, Cambridge MA 02139-4307 USA, Phone: (617) 253-7258, tidor@mit.edu.

†These authors contributed equally to this work

NIH Public Access

Author Manuscript

Mol Biosyst. Author manuscript; available in PMC 2012 November 23.

Published in final edited form as:

Mol Biosyst. 2010 October ; 6(10): 1890–1900. doi:10.1039/b918098b.

$watermark-text

$watermark-text

$watermark-text

Page 2

Constructing an ODE model is comprised of writing kinetic rate equations that describe the

time rate of change of the various chemical species (representing the model topopology),

and determining the unknown parameters in those equations (typically rate constants and

initial concentrations). Unknown parameters are estimated from a variety of data that often

includes time-course measurements of concentration or activity. In this study, we have

focused on the estimation of parameters, which is often referred to as model calibration.

Using computational modeling and experimental design methodology, we have found that

the selection of a set of experiments whose members provide complementary information

can lead to efficient model calibration.

It should be noted that the problem of model calibration is different from model

construction, where increasing numbers of parameters can be used to improve the fit to any

given set of measurements, although parameter uncertainty may remain large. There is a

considerable body of work focused on the problem of model complexity as it relates to

parameter uncertainty.17-20 In general these methods attempt to balance the ability of a more

complex model to reduce fitting errors against the increased likelihood that a more

complicated model will be able to fit the data by chance. Here we fix the model structure

and number of parameters and vary only the measurements taken to develop a strategy for

fitting the constant number of parameters with as little uncertainty as possible.

A detailed treatment of the theory for the current study is present in the Theory section. Here

we provide a framework treatment of that theory. The quality of fit between measurements

and a model can be expressed as the weighted sum-of-squares of the disagreement between

them, which is a chi-squared (χ2) metric. Finding parameter values for a fixed model

topology that minimizes χ2 gives the best-fit parameter values, but because of measurement

uncertainty, different sets of parameter values may be consistent with any given set of

measurements. A common approximation of this parameter uncertainty is to expand χ2 as a

function of the parameters and truncate after the second-order term. The linear term vanishes

because the first derivative of χ2 with respect to parameters is zero when the expansion is

carried out about the best-fit parameter values. χ2 is thus approximated as a constant plus

the second-order terms involving the second derivative of the χ2 quality of fit with respect

to the parameters, known as the Hessian. A given amount of measurement uncertainty leads

to an ellipsoid shaped envelope of constant χ2 in an appropriately scaled parameter space.

Sets of parameters within the envelope are consistent with the measurements and their

associated uncertainty.

Longer axes of the ellipsoid correspond to parameter combinations of greater

uncertainty(i.e., that are less well determined by the measurements), whereas shorter

ellipsoidal axes correspond to parameter combinations of less uncertainty. The mathematics

is such that the aces directions of the ellipsoid are given by the eigenvectors (νi’s) of the

Hessian, and their associated uncertainty is given by the reciprocal of the square root of the

corresponding eigenvalues (λi−1/2). Thus, a set of parameters is well determined by a

collection of measurements when the eigenvalues of the corresponding Hessian are all

sufficiently large that they correspond to small relative parameter uncertainty.

Recently Gutenkunst et al.21 examined parameter uncertainty for 17 models in the EMBL

BioModels Database.22 In their study, the authors assumed noise-free measurements of

every model species sampled continuously in time. The study found that the eigenvalues of

the Hessian spanned a large range (> 106). From this they suggested that, while it may be

possible to estimate some parameters from system-wide data, in practice it would be

difficult or impossible to estimate most of the parameters even from an unrealistically high-

quality data set.21,23,24 Moreover, they pointed out that due to the high eccentricity and

skewness of uncertainly ellipses in parameter space, system-wide data can define system

Apgar et al.

Page 2

Mol Biosyst. Author manuscript; available in PMC 2012 November 23.

$watermark-text

$watermark-text

$watermark-text

Page 3

behavior better than independent measurements of each parameter and may also produce

better predictions in some circumstances.

Here we extend the previous work by more fully considering the effect of experimental

perturbations on the parameter estimation problem and use experimental design to probe for

particularly effective perturbation experiments. The χ2 goodness of fit metric depends on

both the model and the set of experimental conditions. Some experiments may be more

helpful in calibrating the model than others. In the current work we use effectively

continuous-time data, but many experiments require the selection of discrete time points for

measurements to be taken.25,26 It is well established in the systems biology literature that

optimal experimental design can have an impact on the parameter estimation problem for a

single experiment.23 ,25,27-29 For example, work by Faller et al. has shown for a small model

of a mitogen activated protein kinase (MAPK) cascade that the application of time-varying

stimulation significantly improved the parameter estimation problem.29 Essentially this

corresponds to finding the time-varying input signal that gives the best shaped error

ellipsoid.

In this work, we apply a related approach and examine the extent to which multiple

complementary experiments can be combined to improve the overall parameter estimation

problem. Figure 1B,C shows the result of combining data from two separate experiments.

The parameter estimates from the individual data sets (blue and red ellipses) tightly

constrain one parameter direction and weakly constrain the other. In Figure 1B, the weakly

constrained parameter directions are very similar, so the parameter estimates from the

combined data set are about the same as the estimates from the individual experiments

(green ellipse); by contrast, in Figure 1C the experiments are complementary and together

dramatically constrain the parameter estimates.

Because complementary experiments can constrain parameter estimation space, we have

developed an approach to identify sets of complementary experiments to optimally minimize

parameter uncertainty and tested it in a pathway model of signaling in response to EGF and

NGF.30 We have selected this model so that our results may be directly compared to the

previously published analysis of this model performed by Gutenkunst et al.21 For

consistency, where possible we have used their methods and formalisms. In selecting sets of

complementary experiments, we have explored a palette of candidate experiments consisting

of overexpression or knockdown of single and multiple genes combined with different doses

of EGF and NGF, either alone or in combination.

Computational experimental design methods determined all 48 free parameters to within

10% of their value using just five complementary experiments. Selection of complementary

experiments was essential, as the same level of model calibration could not be achieved with

arbitrary experiments or even with a larger number of “highly informative” experiments.

Moreover, we argue that predictions that are sensitive to information complementary to that

used to parameterize a model could be significantly in error. Experimental design methods

can provide sufficient coverage for all parameter directions and thus guide model calibration

for a given topology to maximize predictive accuracy. As systems biology models are

applied to target identification and clinical trial design, the use of experimental design

approaches to improve model prediction quality could be of crucial importance.

Previous work on the model calibration problem has focused on optimization within the

scope of a single experiment.31,32 Examples include selecting optimal time points,33,34

species,35-38 or stimulus conditions5,39,40 that would be most effective in reducing

parameter uncertainty. However, even highly optimized single experiments are generally

insufficient for model calibration. For this reason, such methods have largely been applied to

Apgar et al.

Page 3

Mol Biosyst. Author manuscript; available in PMC 2012 November 23.

$watermark-text

$watermark-text

$watermark-text

Page 4

smaller scale problems. The current work is different in spirit in that it addresses the

question of how improved model calibration might result from combinations of experiments

that could collectively define all of the parameters. By design, the individual experiments

may be easier to implement, yet relatively small combinations of simple experiments can

determine all parameters in a medium-sized pathway model.

Theory

In this work we formulate the model calibration problem as a nonlinear least squares

optimization problem, where the goal is to find the set of parameters that minimizes the fit

metric,21

(1)

where nc is the number of experimental conditions, ns is the number of species for which

measurements are available, the indices c and s run over the conditions and species,

respectively, Tc, is the length of the time course for condition c , ys,c(p,t) is the model output

for species s and condition c at time t with parameter set p , ys,c (p*,t) is the corresponding

output for the true model parameterization, and

taken as proportional to the uncertainty of the experimental measurement.

is a weighting factor that is often

There is a significant amount of work devoted to how best to solve this optimization

problem for biological models.41-44 However, in any experimental system, there will always

be uncertainty in the data, which means there will be some range of parameter values that,

while not optimal, cannot be excluded based on the data. Given a maximum acceptable

fitting error, the calibration problem becomes that of finding all parameter sets such

that the error is less than this threshold. In a neighborhood around the optimum

parameterization p *, the least squares cost function can be approximated by its Taylor

series expansion

(2)

Equation 2 describes an np -dimensional ellipsoid in parameter space (np being the number

of fitted parameters), where all of the parameterizations inside the ellipsoid are feasible. The

size and shape of this ellipsoid describe the multidimensional parameter uncertainty. For

example, the longest axis of the ellipsoid corresponds to the parameter direction (that is, the

linear combination of parameters) with the worst error. Likewise, the axis-parallel bounding

box defines the error range for individual parameters. Figure 1 shows an example for a two-

dimensional system. An important distinction illustrated in Figure 1A is that some parameter

directions can have very small uncertainty, while the individual parameters can be quite

uncertain.21 For example, forward and reverse kinetic constants for a binding reaction may

be poorly constrained, yet the equilibrium constant (given by their ratio), can be well

defined.

H, the matrix of second derivatives of the fit metric is known as the Hessian, where Hi, j is

the derivative of χ2(p) with respect to log(pi) and log(pj).

(3)

Apgar et al. Page 4

Mol Biosyst. Author manuscript; available in PMC 2012 November 23.

$watermark-text

$watermark-text

$watermark-text

Page 5

The derivative is taken with respect to the natural logarithm of pi to obtain a relative

uncertainty. We can dissect the parameter uncertainty in terms of the eigenvalues λi and

eigenvectors νi of H. The eigenvectors form a natural coordinate system for the ellipsoid,

pointing along the axes. The lengths of the axes are proportional to

smaller eigenvalues correspond to larger relative parameter error.41

, meaning that

Results

Experimental design and computer simulations were applied in tandem to select a collection

of experiments that together could most directly establish each of the rate constant

parameters for the EGF/NGF signaling pathway modeled here. To define all 48 rate

parameters, experimental design procedures must select a set of experiments that together

exercise the model in complementary and sufficiently different ways, rather than simply

choosing multiple different experiments that exercise the model in similar ways. For this

work we chose a palette of experimental perturbations consisting of stimulation with EGF

(107, 105, 103, 10, or 0 molecules/cell-volume) or NGF (4.52×107, 4.52×105, 4.52×103,

45.2, or 0 molecules/cell-volume) individually, or combined treatment with both ligands.

We supplemented this choice of ligand stimulation with a panel of experiments in which

protein expression levels could be modulated by 100-fold overexpression or knockdown for

individual proteins in the network. We then constructed candidate experiments from the

combination of ligand choice and protein expression level changes; specifically, each

experiment was allowed to comprise one stimulation pattern and up to three simultaneous

changes in protein expression level. This experimental set-up resulted in a trial perturbation

set of 164,500 individual computational experiments.

All parameters can be determined to high accuracy

The experiments were evaluated and the number of rate parameter directions determined to

within 10% of their nominal value for each experiment was recorded. The best individual

experiment in this set defined only 29 of the 48 rate parameter directions to this high level of

accuracy. In order to improve on this result, each single experiment was re-evaluated to

determine how many new rate parameter directions could be defined to within 10% of their

nominal value when combined with the best individual experiment. In this manner, a greedy

algorithm was applied to select sequentially sets of experiments based on the ability to

generate tighter bounds on parameter estimates. The results of this greedy algorithm are

shown in Figure 2A. The parameter uncertainties are expressed as an eigenspectrum for each

set of experiments, with increasingly larger experimental sets displayed along the abscissa

and eigenvalues displayed along the ordinate. The horizontal dashed line indicates the 10%

error level, and the number of eigenvalues above the dashed line represents the number of

parameter directions determined to the 10% error level. In Figure 2D, the number of

parameters estimated to the 10% level is shown as a function of the number of experiments

within the experimental set. It is striking to observe that, by properly choosing the correct

combination of experiments, only five total experiments are sufficient to determine all 48

directions (and, indeed, all 48 actual parameters) to within 10% accuracy. This result

indicates that parameter uncertainty, rather than being inherent to biological models, can be

progressively reduced by perturbation experiments.

The five experiments determined here to elucidate all 48 parameters are listed in Table I.

The selected experiments include a tendency for dual stimulation with both EGF and NGF,

combined with multiple protein-expression changes, and a preference for overexpression as

opposed to knockdown of given proteins. Interestingly, the experiments do not appear to

systematically explore all regions of the perturbation space. For instance, four of the five

experiments have a low dose of EGF stimulation, and three of the five experiments have a

Apgar et al.

Page 5

Mol Biosyst. Author manuscript; available in PMC 2012 November 23.

$watermark-text

$watermark-text

$watermark-text