Content uploaded by Michael Backenköhler

Author content

All content in this area was uploaded by Michael Backenköhler on Mar 18, 2021

Content may be subject to copyright.

Analysis of Markov Jump Processes under

Terminal Constraints

Michael Backenk¨ohler1,B, Luca Bortolussi2,3, Gerrit Großmann1, Verena

Wolf1,3

1Saarbr¨ucken Graduate School of Computer Science, Saarland University, Saarland

Informatics Campus E1 3, Saarbr¨ucken, Germany

Bmichael.backenkoehler@uni-saarland.de

2Univeristy of Trieste, Trieste, Italy

3Saarland University, Saarland Informatics Campus E1 3, Saarbr¨ucken, Germany

Abstract. Many probabilistic inference problems such as stochastic ﬁl-

tering or the computation of rare event probabilities require model anal-

ysis under initial and terminal constraints. We propose a solution to

this bridging problem for the widely used class of population-structured

Markov jump processes. The method is based on a state-space lumping

scheme that aggregates states in a grid structure. The resulting approxi-

mate bridging distribution is used to iteratively reﬁne relevant and trun-

cate irrelevant parts of the state-space. This way, the algorithm learns

a well-justiﬁed ﬁnite-state projection yielding guaranteed lower bounds

for the system behavior under endpoint constraints. We demonstrate the

method’s applicability to a wide range of problems such as Bayesian

inference and the analysis of rare events.

Keywords: Bayesian Inference ·Bridging problem ·Smoothing ·Lump-

ing ·Rare Events.

1 Introduction

Discrete-valued continuous-time Markov Jump Processes (MJP) are widely used

to model the time evolution of complex discrete phenomena in continuous time.

Such problems naturally occur in a wide range of areas such as chemistry [16],

systems biology [49,46], epidemiology [36] as well as queuing systems [10] and

ﬁnance [39]. In many applications, an MJP describes the stochastic interaction

of populations of agents. The state variables are counts of individual entities of

diﬀerent populations.

Many tasks, such as the analysis of rare events or the inference of agent

counts under partial observations naturally introduce terminal constraints on

the system. In these cases, the system’s initial state is known, as well as the

system’s (partial) state at a later time-point. The probabilities corresponding

to this so-called bridging problem are often referred to as bridging probabilities

[17,19]. For instance, if the exact, full state of the process Xthas been observed

at time 0 and T, the bridging distribution is given by

Pr(Xt=x|X0=x0, XT=xg)

2 M. Backenk¨ohler et al.

for all states xand times t∈[0, T ]. Often, the condition is more complex, such

that in addition to an initial distribution, a terminal distribution is present.

Such problems typically arise in a Bayesian setting, where the a priori behavior

of a system is ﬁltered such that the posterior behavior is compatible with noisy,

partial observations [11,25]. For example, time-series data of protein levels is

available while the mRNA concentration is not [1,25]. In such a scenario our

method can be used to identify a good truncation to analyze the probabilities

of mRNA levels.

Bridging probabilities also appear in the context of rare events. Here, the rare

event is the terminal constraint because we are only interested in paths contain-

ing the event. Typically researchers have to resort to Monte-carlo simulations in

combination with variance reduction techniques in such cases [14,26].

Eﬃcient numerical approaches that are not based on sampling or ad-hoc

approximations have rarely been developed.

Here, we combine state-of-the-art truncation strategies based on a forward

analysis [28,4] with a reﬁnement approach that starts from an abstract MJP with

lumped states. We base this lumping on a grid-like partitioning of the state-space.

Throughout a lumped state, we assume a uniform distribution that gives an

eﬃcient and convenient abstraction of the original MJP. Note that the lumping

does not follow the classical paradigm of Markov chain lumpability [12] or its

variants [15]. Instead of an approximate block structure of the transition-matrix

used in that context, we base our partitioning on a segmentation of the molecule

counts. Moreover, during the iterative reﬁnement of our abstraction, we identify

those regions of the state-space that contribute most to the bridging distribution.

In particular, we reﬁne those lumped states that have a bridging probability

above a certain threshold δand truncate all other macro-states. This way, the

algorithm learns a truncation capturing most of the bridging probabilities. This

truncation provides guaranteed lower bounds because it is at the granularity of

the original model.

In the rest of the paper, after presenting related work (Section 2) and back-

ground (Section 3), we discuss the method (Section 4) and several applications,

including the computation of rare event probabilities as well as Bayesian smooth-

ing and ﬁltering (Section 5).

2 Related Work

The problem of endpoint constrained analysis occurs in the context of Bayesian

estimation [41]. For population-structured MJPs, this problem has been ad-

dressed by Huang et al. [25] using moment closure approximations and by Wild-

ner and K¨oppl [48] further employing variational inference. Golightly and Sher-

lock modiﬁed stochastic simulation algorithms to approximatively augment gen-

erated trajectories [17]. Since a statistically exact augmentation is only possible

for few simple cases, diﬀusion approximations [18] and moment approximations

[35] have been employed. Such approximations, however, do not give any guaran-

tees on the approximation error and may suﬀer from numerical instabilities [43].

Analysis of Markov Jump Processes under Terminal Constraints 3

The bridging problem also arises during the estimation of ﬁrst passage times

and rare event analysis. Approaches for ﬁrst-passage times are often of heuristic

nature [42,22,8]. Rigorous approaches yielding guaranteed bounds are currently

limited by the performance of state-of-the-art optimization software [6]. In bi-

ological applications, rare events of interest are typically related to the reach-

ability of certain thresholds on molecule counts or mode switching [45]. Most

methods for the estimation of rare event probabilities rely on importance sam-

pling [26,14]. For other queries, alternative variance reduction techniques such

as control variates are available [5]. Apart from sampling-based approaches, dy-

namic ﬁnite-state projections have been employed by Mikeev et al. [34], but are

lacking automated truncation schemes.

The analysis of countably inﬁnite state-spaces is often handled by a pre-

deﬁned truncation [27]. Sophisticated state-space truncations for the (uncondi-

tioned) forward analysis have been developed to give lower bounds and rely on a

trade-oﬀ between computational load and tightness of the bound [37,28,4,24,31].

Reachability analysis, which is relevant in the context of probabilistic veri-

ﬁcation [8,38], is a bridging problem where the endpoint constraint is the visit

of a set of goal states. Backward probabilities are commonly used to compute

reachability likelihoods [2,50]. Approximate techniques for reachability, based

on moment closure and stochastic approximation, have also been developed in

[8,9], but lack error guarantees. There is also a conceptual similarity between

computing bridging probabilities and the forward-backward algorithm for com-

puting state-wise posterior marginals in hidden Markov models (HMMs) [40].

Like MJPs, HMMs are a generative model that can be conditioned on obser-

vations. We only consider two observations (initial and terminal state) that are

not necessarily noisy but the forward and backward probabilities admit the same

meaning.

3 Preliminaries

3.1 Markov Jump Processes with Population Structure

A population-structured Markov jump process (MJP) describes the stochastic

interactions among agents of distinct types in a well-stirred reactor. The assump-

tion of all agents being equally distributed in space, allows to only keep track

of the overall copy number of agents for each type. Therefore the state-space is

S ⊆ NnSwhere nSdenotes the number of agent types or populations. Interac-

tions between agents are expressed as reactions. These reactions have associated

gains and losses of agents, given by non-negative integer vectors v−

jand v+

jfor

reaction j, respectively. The overall eﬀect is given by vj=v+

j−v−

j. A reaction

between agents of types S1, . . . , SnSis speciﬁed in the following form:

nS

X

`=1

v−

j` S`

αj(x)

−−−→

nS

X

`=1

v+

j` S`.(1)

4 M. Backenk¨ohler et al.

The propensity function αjgives the rate of the exponentially distributed ﬁring

time of the reaction as a function of the current system state x∈ S. In population

models, mass-action propensities are most common. In this case the ﬁring rate

is given by the product of the number of reactant combinations in xand a rate

constant cj, i.e.

αj(x):=cj

nS

Y

`=1 x`

v−

j` .(2)

In this case, we give the rate constant in (1) instead of the function αj. For a

given set of nRreactions, we deﬁne a stochastic process {Xt}t≥0describing the

evolution of the population sizes over time t. Due to the assumption of exponen-

tially distributed ﬁring times, Xis a continuous-time Markov chain (CTMC) on

Swith inﬁnitesimal generator matrix Q, where the entries of Qare

Qx,y =(Pj:x+vj=yαj(x),if x6=y,

−PnR

j=1 αj(x),otherwise. (3)

The probability distribution over time can be analyzed as an initial value prob-

lem. Given an initial state x0, the distribution1

π(xi, t) = Pr(Xt=xi|X0=x0), t ≥0 (4)

evolves according to the Kolmogorov forward equation

d

dtπ(t) = π(t)Q , (5)

where π(t) is an arbitrary vectorization (π(x1, t), π(x2, t), . . . , π(x|S |, t)) of the

states.

Let xg∈ S be a ﬁxed goal state. Given the terminal constraint Pr(XT=xg)

for some T≥0, we are interested in the so-called backward probabilities

β(xi, t) = Pr(XT=xg|Xt=xi), t ≤T . (6)

Note that β(·, t) is a function of the conditional event and thus is no probability

distribution over the state-space. Instead β(·, t) gives the reaching probabilities

for all states over the time span of [t, T ]. To compute these probabilities, we can

employ the Kolmogorov backward equation

d

dtβ(t) = Qβ(t)>,(7)

where we use the same vectorization to construct β(t) as we used for π(t). The

above equation is integrated backwards in time and yields the reachability prob-

ability for each state xiand time t<T of ending up in xgat time T.

1In the sequel, xidenotes a state with index iinstead of its i-th component.

Analysis of Markov Jump Processes under Terminal Constraints 5

The state-space of many MJPs with population structure, even simple ones,

is countably inﬁnite. In this case, we have to truncate the state-space to a rea-

sonable ﬁnite subset. The choice of this truncation heavily depends on the goal of

the analysis. If one is interested in the most “common” behavior, for example, a

dynamic mass-based truncation scheme is most appropriate [32]. Such a scheme

truncates states with small probability during the numerical integration. How-

ever, common mass-based truncation schemes are not as useful for the bridging

problem. This is because trajectories that meet the speciﬁc terminal constraints

can be far oﬀ the main bulk of the probability mass. We solve this problem by

a state-space lumping in connection with an iterative reﬁnement scheme.

Consider as an example a birth-death process. This model can be used to

model a wide variety of phenomena and often constitutes a sub-module of larger

models. For example, it can be interpreted as an M/M/1 queue with service rates

being linearly dependent on the queue length. Note, that even for this simple

model, the state-space is countably inﬁnite.

Model 1 (Birth-Death Process). The model consists of exponentially dis-

tributed arrivals and service times proportional to queue length. It can be ex-

pressed using two mass-action reactions:

∅10

−→ Xand X.1

−→ ∅.

The initial condition X0= 0 holds with probability one.

3.2 Bridging Distribution

The process’ probability distribution given both initial and terminal constraints

is formally described by the conditional probabilities

γ(xi, t) = Pr(Xt=xi|X0=x0, XT=xg),0≤t≤T(8)

for ﬁxed initial state x0and terminal state xg. We call these probabilities the

bridging probabilities. It is straight-forward to see that γadmits the factorization

γ(xi, t) = π(xi, t)β(xi, t)/π(xg, T ) (9)

due to the Markov property. The normalization factor, given by the reachability

probability π(xg, T ) = β(x0,0), ensures that γ(·, t) is a distribution for all time

points t∈[0, T ]. We call each γ(·, t) a bridging distribution. From the Kolmogorov

equations (5) and (7) we can obtain both the forward probabilities π(·, t) and

the backward probabilities β(·, t) for t<T.

We can easily extend this procedure to deal with hitting times constrained

by a ﬁnite time-horizon by making the goal state xgabsorbing.

In Figure 1 we plot the forward, backward, and bridging probabilities for

Model 1. The probabilities are computed on a [0,100] state-space truncation. The

approximate forward solution ˆπshows how the probability mass drifts upwards

towards the stationary distribution Poisson(100). The backward probabilities

6 M. Backenk¨ohler et al.

Fig. 1. Forward, backward, and bridging probabilities for Model 1 with initial con-

straint X0= 0 and terminal constraint X10 = 40 on a truncated state-space. Proba-

bilities over 0.1 in ˆπand ˆ

βare given full intensity for visual clarity. The lightly shaded

area (≥60) indicates a region being more relevant for the forward than for the bridging

probabilities.

are highest for states below the goal state xg= 40. This is expected because

upwards drift makes reaching xgmore probable for “lower” states. Finally, the

approximate bridging distribution ˆγcan be recognized to be proportional to the

product of forward ˆπand backward probabilities ˆ

β.

4 Bridge Truncation via Lumping Approximations

We ﬁrst discuss the truncation of countably inﬁnite state-spaces to analyze back-

ward and forward probabilities (Section 4.1). To identify eﬀective truncations we

employ a lumping scheme. In Section 4.2, we explain the construction of macro-

states and assumptions made, as well as the eﬃcient calculation of transition

rates between them. Finally, in Section 4.3 we present an iterative reﬁnement

algorithm yielding a suitable truncation for the bridging problem.

4.1 Finite State Projection

Even in simple models such as a birth-death Process (Model 1), the reachable

state-space is countably inﬁnite. Direct analyzes of backward (6) and forward

equations (4) are often infeasible. Instead, the integration of these diﬀerential

equations requires working with a ﬁnite subset of the inﬁnite state-space [37]. If

states are truncated, their incoming transitions from states that are not trun-

cated can be re-directed to a sink state. The accumulated probability in this

sink state is then used as an error estimate for the forward integration scheme.

Consequently, many truncation schemes, such as dynamic truncations [4], aim

to minimize the amount of “lost mass” of the forward probability. We use the

same truncation method but base the truncation on bridging probabilities rather

than the forward probabilities.

4.2 State-Space Lumping

When dealing with bridging problems, the most likely trajectories from the initial

to the terminal state are typically not known a priori. Especially if the event in

Analysis of Markov Jump Processes under Terminal Constraints 7

question is rare, obtaining a state-space truncation adapted to its constraints is

diﬃcult. We devise a lumping scheme that groups nearby states, i.e. molecule

counts, into larger macro-states. A macro-state is a collection of states treated

as one state in a lumped model, which can be seen as an abstraction of the

original model. These macro-states form a partitioning of the state-space. In this

lumped model, we assume a uniform distribution over the constituent micro-

states inside each macro-state. Thus, given that the system is in a particular

macro-state, all of its micro-states are equally likely. This partitioning allows us

to analyze signiﬁcant regions of the state-space eﬃciently albeit under a rough

approximation of the dynamics. Iterative reﬁnement of the state-space after each

analysis moves the dynamics closer to the original model. In the ﬁnal step of the

iteration, the considered system states are at the granularity of the original model

such that no approximation error is introduced by assumptions of the lumping

scheme. Computational eﬃciency is retained by truncating in each iteration

step those states that contribute little probability mass to the (approximated)

bridging distributions.

We choose a lumping scheme based on a grid of hypercube macro-states whose

endpoints belong to a predeﬁned grid. This topology makes the computation

of transition rates between macro-states particularly convenient. Mass-action

reaction rates, for example, can be given in a closed-form due to the Faulhaber

formulae. More complicated rate functions such as Hill functions can often be

handled as well by taking appropriate integrals.

Our choice is a scheme that uses nS-dimensional hypercubes. A macro-state

¯xi(`(i), u(i)) (denoted by ¯xifor notational ease) can therefore be described by

two vectors `(i)and u(i). The vector `(i)gives the corner closest to the origin,

while u(i)gives the corner farthest from the origin. Formally,

¯xi= ¯xi(`(i), u(i)) = {x∈NnS|`(i)≤x≤u(i)},(10)

where ’≤’ stands for the element-wise comparison. This choice of topology makes

the computation of transition rates between macro-states particularly conve-

nient: Suppose we are interested in the set of micro-states in macro-state ¯xithat

can transition to macro-state ¯xkvia reaction j. It is easy to see that this set is

itself an interval-deﬁned macro-state ¯xij

−→k. To compute this macro-state we can

simply shift ¯xiby vj, take the intersection with ¯xkand project this set back.

Formally,

¯xij

−→k= ((¯xi+vj)∩¯xk)−vj,(11)

where the additions are applied element-wise to all states making up the macro-

states. For the correct handling of the truncation it is useful to deﬁne a general

exit state

¯xij

−→ = ((¯xi+vj)\¯xi)−vj.(12)

This state captures all micro-states inside ¯xithat can leave the state via reaction

j. Note that all operations preserve the structure of a macro-state as deﬁned in

(10). Since a macro-state is based on intervals the computation of the transition

rate is often straight-forward. Under the assumption of polynomial rates, as

8 M. Backenk¨ohler et al.

Fig. 2. A lumping approximation of Model 1 on the state-space truncation to [0,200]

on t∈[0,50]. On the left-hand side solutions of a regular truncation approximation

and a lumped truncation (macro-state size is 5) are given. On the right-hand side the

respective terminal distributions Pr(X50 =xi) are contrasted.

it is the case for mass-action systems, we can compute the sum of rates over

this transition set eﬃciently using Faulhaber’s formula. We deﬁne the lumped

transition function

¯αj(¯x) = X

x∈¯x

αj(x) (13)

for macro-state ¯xand reaction j. As an example consider the following mass-

action reaction 2Xc

−→ ∅.For macro-state ¯x={0, . . . , n}we can compute the

corresponding lumped transition rate

¯α(¯x) = c

2

n

X

i=1

i(i−1) = c

2

n

X

i=1

(i2−i) = c

22n3+ 3n2+n

6−n2+n

2

eliminating the explicit summation in the lumped propensity function.

For polynomial propensity functions αsuch formulae are easily obtained au-

tomatically. For non-polynomial propensity functions, we can use the continuous

integral as an approximation. This is demonstrated on a case study in Section 5.2.

Using the transition set computation (11) and the lumped propensity func-

tion (13) we can populate the Q-matrix of the ﬁnite lumping approximation:

¯

Q¯xi,¯xk=

PnR

j=1 ¯αj¯xij

−→k/vol (¯xi),if ¯xi6= ¯xk

−PnR

j=1 ¯αj¯xij

−→/vol (¯xi),otherwise

(14)

In addition to the lumped rate function over the transition state ¯xij

−→k, we need to

divide by the total volume of the lumped state ¯xi. This is due to the assumption

of a uniform distribution inside the macro-states. Using this Q-matrix, we can

compute the forward and backward solution using the respective Kolmogorov

equations (5) and (7).

Interestingly, the lumped distribution tends to be less concentrated. This is

due to the assumption of a uniform distribution inside macro-states. This eﬀect

Analysis of Markov Jump Processes under Terminal Constraints 9

is illustrated by the example of a birth-death process in Figure 2. Due to this

eﬀect, an iterative reﬁnement typically keeps an over-approximation in terms of

state-space area. This is a desirable feature since relevant regions are less likely

to be pruned due to lumping approximations.

4.3 Iterative Reﬁnement Algorithm

The iterative reﬁnement algorithm (Alg. 1) starts with a set of large macro-states

that are iteratively reﬁned, based on approximate solutions to the bridging prob-

lem. We start by constructing square macro-states of size 2min each dimension

for some m∈Nsuch that they form a large-scale grid S(0). Hence, each initial

macro-state has a volume of (2m)nS. This choice of grid size is convenient be-

cause we can halve states in each dimension. Moreover, this choice ensures that

all states have equal volume and we end up with states of volume 20= 1 which

is equivalent to a truncation of the original non-lumped state-space.

An iteration of the state-space reﬁnement starts by computing both the for-

ward and backward probabilities (lines 2 and 3) via integration of (5) and (7),

respectively, using the lumped ˆ

Q-matrix. Based on the resulting approximate

forward and backward probabilities, we compute an approximation of the bridg-

ing distributions (line 4). This is done for each time-point in an equispaced grid

on [0, T ]. The time grid granularity is a hyper-parameter of the algorithm. If

the grid is too ﬁne, the memory overhead of storing backward ˆ

β(i)and forward

solutions ˆπ(i)increases.2If, on the other hand, the granularity is too low, too

much of the state-space might be truncated. Based on a threshold parameter

δ > 0 states are either removed or split (line 7), depending on the mass assigned

to them by the approximate bridging probabilities ˆγ(i)

t. A state can be split by

the split-function which halves the state in each dimension. Otherwise, it is

removed. Thus, each macro-state is either split into 2nSnew states or removed

entirely. The result forms the next lumped state-space S(i+1). The Q-matrix is

adjusted (line 10) such that transition rates for S(i+1) are calculated accord-

ing to (14). Entries of truncated states are removed from the transition matrix.

Transitions leading to them are re-directed to a sink state (see Section 4.1). Af-

ter miterations (we started with states of side lengths 2m) we have a standard

ﬁnite state projection scheme on the original model tailored to computing an

approximation of the bridging distribution.

In Figure 3 we give a demonstration of how Algorithm 1 works to reﬁne the

state-space iteratively. Starting with an initial lumped state-space S(0) covering

a large area of the state-space, repeated evaluations of the bridging distributions

are performed. After ﬁve iterations the remaining truncation includes all states

that signiﬁcantly contribute to the bridging probabilities over the times [0, T ].

It is important to realize that determining the most relevant states is the

main challenge. The above algorithm solves this problem by considering only

2We denote the approximations with a hat (e.g. ˆπ) rather than a bar (e.g. ¯π) to

indicate that not only the lumping approximation but also a truncation is applied

and similarly for the Q-matrix.

10 M. Backenk¨ohler et al.

Algorithm 1: Iterative reﬁnement for the bridging problem

input : Initial partitioning S(0) , truncation threshold δ

output: approximate bridging distribution ˆγ

1for i= 1,...,m do

2ˆπ(i)

t←solve approximate forward equation on S(i);

3ˆ

β(i)

t←solve approximate backward equation on S(i);

4ˆγ(i)

t←ˆ

β(i)ˆπ(i)/ˆπ(xg, T ); /* approximate bridging distribution */

5S(i+1) ← ∅;

6foreach ¯x∈ S(i)do

7if ∃t.ˆγ(i)

t(¯x)≥δ;/* refine based on bridging probabilities */

8then

9S(i+1) ← S(i+1) ∪split( ¯x);

10 update ˆ

Q-matrix;

11 return ˆγ(i);

Fig. 3. The state-space reﬁnement algorithm on two parallel unit-rate arrival processes.

The bridging problem from (0,0) to (64,64) and T= 10 and truncation threshold

δ= 5e-3. States with a bridging probability below δare light grey. The macro-state

containing the goal state is marked in black. The initial macro-states are of size 16×16.

those parts of the state-space that contribute most to the bridging probabilities.

The truncation is tailored to this condition and might ignore regions that are

likely in the unconditioned case. For instance, in Fig. 1 the bridging probabili-

ties mostly remain below a population threshold of #X= 60 (as indicated by

the lighter/darker coloring), while the forward probabilities mostly exceed this

bound. Hence, in this example a signiﬁcant portion of the forward probabilities

ˆπ(i)

tis captured by the sink state. However, the condition in line 7 in Algorithm 1

ensures that states contributing signiﬁcantly to ˆγ(i)

twill be kept and reﬁned in

the next iteration.

5 Results

We present four examples in this section to evaluate our proposed method.

A prototype was implemented in Python 3.8. For numerical integration we

Analysis of Markov Jump Processes under Terminal Constraints 11

threshold δ1e-2 1e-3 1e-4 1e-5

truncation size 1154 2354 3170 3898

overall states 2074 3546 4586 5450

estimate 8.8851e-30 1.8557e-29 1.8625e-29 1.8625e-29

rel. error 5.2297e-01 3.6667e-03 3.7423e-05 9.5259e-08

Table 1. Estimated reachability probabilities based on varying truncation thresholds

δ: The true probability is 1.8625e-29. We also report the size of the ﬁnal truncation and

the accumulated size of all truncations during reﬁnement iterations (overall states).

used the Scipy implementation [47] of the implicit method based on backward-

diﬀerentiation formulas [13]. The analysis as a Jupyter notebook is made avail-

able online3.

5.1 Bounding Rare Event Probabilities

We consider a simple model of two parallel Poisson processes describing the

production of two types of agents. The corresponding probability distribution

has Poisson product form at all time points t≥0 and hence we can compare

the accuracy of our numerical results with the exact analytic solution. We use

the proposed approach to compute lower bounds for rare event probabilities. 4

Model 2 (Parallel Poisson Processes). The model consists of two parallel

independent Poisson processes with unit rates.

∅1

−→ Aand ∅1

−→ B .

The initial condition X0= (0,0) holds with probability one. After ttime units

each species abundance is Poisson distributed with rate λ=t.

We consider the ﬁnal constraint of reaching a state where both processes exceed

a threshold of 64 at time 20. Without prior knowledge, a reasonable truncation

would have been 160×160. But our analysis shows that just 20% of the states are

necessary to capture over 99.6% of the probability mass reaching the target event

(cf. Table 1). Decreasing the threshold δleads to a larger set of states retained

after truncation as more of the bridging distribution is included (cf. Figure 4).

We observe an increase in truncation size that is approximately logarithmic in δ,

which, in this example, indicates robustness of the method with respect to the

choice of δ.

3https://www.github.com/mbackenkoehler/mjp bridging

4These bounds are rigorous up to the approximation error of the numerical inte-

gration scheme. However, the forward solution could be replaced by an adaptive

uniformization approach [3] for a more rigorous integration error control.

12 M. Backenk¨ohler et al.

Fig. 4. State-space truncation for varying values of the threshold parameter δ: Two

parallel Poisson processes under terminal constraints X(A)

20 ≥64 and X(B)

20 ≥64. The

initial macro-states are 16 ×16 such that the ﬁnal states are regular micro states.

Comparison to other methods The truncation approach that we apply is similar

to the one used by Mikeev et al. [34] for rare event estimation. However, they used

a given linearly biased MJP model to obtain a truncation. A general strategy

to compute an appropriate biasing was not proposed. It is possible to adapt

our truncation approach to the dynamic scheme in Ref. [34] where states are

removed in an on-the-ﬂy fashion during numerical integration.

A ﬁnite state-space truncation covering the same area as the initial lumping

approximation would contain 25,600 states.5The standard approach would be

to build up the entire state-space for such a model [27]. Even using a conser-

vative truncation threshold δ= 1e-5, our method yields an accurate estimate

using only about a ﬁfth (5450) of this accumulated over all intermediate lumped

approximations.

5.2 Mode Switching

Mode switching occurs in models exhibiting multi-modal behavior [44] when a

trajectory traverses a potential barrier from one mode to another. Often, mode

switching is a rare event and occurs in the context of gene regulatory networks

where a mode is characterized by the set of genes being currently active [30].

Similar dynamics also commonly occur in queuing models where a system may

for example switch its operating behavior stochastically if traﬃc increases above

or decreases below certain thresholds. Using the presented method, we can get

both a qualitative and quantitative understanding of switching behavior without

resorting to Monte-Carlo methods such as importance sampling.

Exclusive Switch The exclusive switch [7] has three diﬀerent modes of opera-

tion, depending on the DNA state, i.e. on whether a protein of type one or two

is bound to the DNA.

5Here, the goal is not treated as a single state. Otherwise, it consists of 24,130 states.

Analysis of Markov Jump Processes under Terminal Constraints 13

Model 3 (Exclusive Switch). The exclusive switch model consists of a pro-

moter region that can express both proteins P1and P2. Both can bind to the

region, suppressing the expression of the other protein. For certain parameteri-

zations, this leads to a bi-modal or even tri-modal behavior.

Dρ

−→ D+P1Dρ

−→ D+P2P1

λ

−→ ∅P2

λ

−→ ∅

D+P1

β

−→ D.P1D.P 1γ

−→ D+P1D.P1

α

−→ D.P1+P1

D+P2

β

−→ D.P2D.P 2γ

−→ D+P2D.P2

α

−→ D.P2+P2

The parameter values are ρ=1e-1, λ=1e-3, β=1e-2, γ=8e-3, and α=1e-1.

Since we know a priori of the three distinct operating modes, we adjust the

method slightly: The state-space for the DNA states is not lumped. Instead

we “stack” lumped approximations of the P1-P2phase space upon each other.

Special treatment of DNA states is common for such models [28].

To analyze the switching, we choose the transition from (variable order: P1,

P2,D,D.P1,D.P2)x1= (32,0,0,0,1) to x2= (0,32,0,1,0) over the time

interval t∈[0,10]. The initial lumping scheme covers up to 80 molecules of P1

and P2for each mode. Macro-states have size 8×8 and the truncation threshold

is δ= 1e-4.

In the analysis of biological switches, not only the switching probability but

also the switching dynamics is a central part of understanding the underlying

biological mechanisms. In Figure 5 (left), we therefore plot the time-varying

probabilities of the gene state conditioned on the mode. We observe a rapid un-

binding of P2, followed by a slow increase of the binding probability for P1. These

dynamics are already qualitatively captured by the ﬁrst lumped approximation

(dashed lines).

Toggle Switch Next, we apply our method to a toggle switch model exhibiting

non-polynomial rate functions. This well-known model considers two proteins A

and Binhibiting the production of the respective other protein [29].

Model 4. Toggle Switch (Hill functions) We have population types Aand B

with the following reactions and reaction rates.

∅α1(·)

−−−→ A , where α1(x) = ρ

1 + xB

, A λ

−→ ∅

∅α1(·)

−−−→ B , where α1(x) = ρ

1 + xA

, B λ

−→ ∅

The parameterization is ρ= 10,λ= 0.1.

Due to the non-polynomial rate functions α1and α2, the transition rates between

macro-states are approximated by using the continuous integral

¯α1(¯x)≈Zb+0.5

a−0.5

ρ

1 + xdx =ρ(log (b+ 1.5) −log (a+ 0.5))

14 M. Backenk¨ohler et al.

Fig. 5. (left) Mode probabilities of the exclusive switch bridging problem over time for

the ﬁrst lumped approximation (dashed lines) and the ﬁnal approximation (solid lines)

with constraints X0= (32,0,0,1,0) and X10 = (0,32,0,0,1). (right) The expected

occupation time (excluding initial and terminal states) for the switching problem of

the toggle switch using Hill-type functions. The bridging problem is from initial (0,120)

to a ﬁrst passage of (120,0) in t∈[0,10].

for a macro-state ¯x={a, . . . , b}.

We analyze the switching scenario from (0,120) to the ﬁrst visit of state

(120,0) up to time T= 10. The initial lumping scheme covers up to 352 molecules

of Aand Band macro-states have size 32 ×32. The truncation threshold is

δ= 1e-4. The resulting truncation is shown in Figure 5 (right). It also illustrates

the kind of insights that can be obtained from the bridging distributions. For

an overview of the switching dynamics, we look at the expected occupation

time under the terminal constraint of having entered state (120,0). Letting the

corresponding hitting time be τ= inf {t≥0|Xt= (120,0)}, the expected

occupation time for some state xis ERτ

01=x(Xt)dt |τ≤10. We observe that

in this example the switching behavior seems to be asymmetrical. The main mass

seems to pass through an area where initially a small number of Amolecules is

produced followed by a total decay of Bmolecules.

5.3 Recursive Bayesian Estimation

We now turn to the method’s application in recursive Bayesian estimation. This

is the problem of estimating the system’s past, present, and future behavior un-

der given observations. Thus, the MJP becomes a hidden Markov model (HMM).

The observations in such models are usually noisy, meaning that we cannot infer

the system state with certainty.

This estimation problem entails more general distributional constraints on

terminal β(·, T ) and initial π(·,0) distributions than the point mass distributions

considered up until now. We can easily extend the forward and backward proba-

bilities to more general initial distributions and terminal distributions β(T). For

Analysis of Markov Jump Processes under Terminal Constraints 15

the forward probabilities we get

π(xi, t) = X

j

Pr(Xt=xi|X0=xj)π(xj,0),(15)

and similarly the backward probabilities are given by

β(xi, t) = X

j

Pr(XT=xj|Xt=xi)βT(xj).(16)

We apply our method to an SEIR (susceptible-exposed-infected-removed) model.

This is widely used to describe the spreading of an epidemic such as the current

COVID-19 outbreak [23,20]. Temporal snapshots of the epidemic spread are

mostly only available for a subset of the population and suﬀer from inaccuracies

of diagnostic tests. Bayesian estimation can then be used to infer the spreading

dynamics given uncertain temporal snapshots.

Model 5 (Epidemics Model). A population of susceptible individuals can

contract a disease from infected agents. In this case, they are exposed, mean-

ing they will become infected but cannot yet infect others. After being infected,

individuals change to the removed state. The mass-action reactions are as fol-

lows.

S+Iλ

−→ E+I E µ

−→ I I ρ

−→ R

The parameter values are λ= 0.5,µ= 3,ρ= 3. Due to the stoichiometric

invariant X(S)

t+X(E)

t+X(I)

t+X(R)

t= const., we can eliminate Rfrom the

system.

We consider the following scenario: We know that initially (t= 0) one in-

dividual is infected and the rest is susceptible. At time t= 0.3 all individuals

are tested for the disease. The test, however, only identiﬁes infected individuals

with probability 0.99. Moreover, the probability of a false positive is 0.05. We

like to identify the distribution given both the initial state and the measurement

at time t= 0.3. In particular, we want to infer the distribution over the latent

counts of Sand Eby recursive Bayesian estimation.

The posterior for nIinfected individuals at time t, given measurement Yt=

ˆnIcan be computed using Bayes’ rule

Pr(X(I)

t=nI|Yt= ˆnI)∝Pr(Yt= ˆnI|X(I)

t=nI) Pr(X(I)

t=nI).(17)

This problem is an extension of the bridging problem discussed up until now.

The diﬀerence is that the terminal posterior is estimated it using the result of the

lumped forward equation and the measurement distribution using (17). Based

on this estimated terminal posterior, we compute the bridging probabilities and

reﬁne the truncation tailored to the location of the posterior distribution. In Fig-

ure 6 (left), we illustrate the bridging distribution between the terminal posterior

and initial distribution. In the context of ﬁltering problems this is commonly re-

ferred to as smoothing. Using the learned truncation, we can obtain the posterior

distribution for the number of infected individuals at t= 0.3 (Figure 6 (middle)).

Moreover, can we infer a distribution over the unknown number of susceptible

and exposed individuals (Figure 6 (right)).

16 M. Backenk¨ohler et al.

Fig. 6. (left) A comparison of the prior dynamics and the posterior smoothing (bridg-

ing) dynamics. (middle) The prior, likelihood, and posterior of the number of infected

individuals nIat time t= 0.3 given the measurement ˆnI= 30. (right) The prior and

posterior distribution over the latent types Eand S.

6 Conclusion

The analysis of Markov Jump processes with constraints on the initial and ter-

minal behavior is an important part of many probabilistic inference tasks such

as parameter estimation using Bayesian or maximum likelihood estimation, in-

ference of latent system behavior, the estimation of rare event probabilities, and

reachability analysis for the veriﬁcation of temporal properties. If endpoint con-

straints correspond to atypical system behaviors, standard analysis methods fail

as they have no strategy to identify those parts of the state-space relevant for

meeting the terminal constraint.

Here, we proposed a method that is not based on stochastic sampling and

statistical estimation but provides a direct numerical approach. It starts with an

abstract lumped model, which is iteratively reﬁned such that only those parts of

the model are considered that contribute to the probabilities of interest. In the

ﬁnal step of the iteration, we operate at the granularity of the original model

and compute lower bounds for these bridging probabilities that are rigorous up

to the error of the numerical integration scheme.

Our method exploits the population structure of the model, which is present

in many important application ﬁelds of MJPs. Based on experience with other

work based on truncation, the approach can be expected to scale up to at least

a few million states [33]. Compared to previous work, our method neither relies

on approximations of unknown accuracy nor additional information such as a

suitable change of measure in the case of importance sampling. It only requires

a truncation threshold and an initial choice for the macro-state sizes.

In future work, we plan to extend our method to hybrid approaches, in which

a moment representation is employed for large populations while discrete counts

are maintained for small populations. Moreover, we will apply our method to

model checking where constraints are described by some temporal logic [21].

Analysis of Markov Jump Processes under Terminal Constraints 17

Acknowledgements This project was supported by the DFG project MULTI-

MODE and Italian PRIN project SEDUCE.

References

1. Adan, A., Alizada, G., Kiraz, Y., Baran, Y., Nalbant, A.: Flow cytometry: basic

principles and applications. Critical reviews in biotechnology 37(2), 163–176 (2017)

2. Amparore, E.G., Donatelli, S.: Backward solution of Markov chains and Markov re-

generative processes: Formalization and applications. Electron. Notes Theor. Com-

put. Sci. 296, 7–26 (2013)

3. Andreychenko, A., Crouzen, P., Mikeev, L., Wolf, V.: On-the-ﬂy uniformiza-

tion of time-inhomogeneous inﬁnite Markov population models. arXiv preprint

arXiv:1006.4425 (2010)

4. Andreychenko, A., Mikeev, L., Spieler, D., Wolf, V.: Parameter identiﬁcation for

Markov models of biochemical reactions. In: International Conference on Computer

Aided Veriﬁcation. pp. 83–98. Springer (2011)

5. Backenk¨ohler, M., Bortolussi, L., Wolf, V.: Control variates for stochastic sim-

ulation of chemical reaction networks. In: Bortolussi, L., Sanguinetti, G. (eds.)

Computational Methods in Systems Biology. pp. 42–59. Springer, Cham (2019)

6. Backenk¨ohler, M., Bortolussi, L., Wolf, V.: Bounding mean ﬁrst passage times in

population continuous-time Markov chains. To appear in Proc. of QEST’20 (2020)

7. Barzel, B., Biham, O.: Calculation of switching times in the genetic toggle switch

and other bistable systems. Physical Review E 78(4), 041919 (2008)

8. Bortolussi, L., Lanciani, R.: Stochastic approximation of global reachability prob-

abilities of Markov population models. In: Computer Performance Engineering -

11th European Workshop, EPEW 2014, Florence, Italy, September 11-12, 2014.

Proceedings. pp. 224–239 (2014)

9. Bortolussi, L., Lanciani, R., Nenzi, L.: Model checking markov population models

by stochastic approximations. Inf. Comput. 262, 189–220 (2018)

10. Breuer, L.: From Markov jump processes to spatial queues. Springer Science &

Business Media (2003)

11. Broemeling, L.D.: Bayesian Inference for Stochastic Processes. CRC Press (2017)

12. Buchholz, P.: Exact and ordinary lumpability in ﬁnite Markov chains. Journal of

applied probability pp. 59–75 (1994)

13. Byrne, G.D., Hindmarsh, A.C.: A polyalgorithm for the numerical solution of ordi-

nary diﬀerential equations. ACM Transactions on Mathematical Software (TOMS)

1(1), 71–96 (1975)

14. Daigle Jr, B.J., Roh, M.K., Gillespie, D.T., Petzold, L.R.: Automated estimation of

rare event probabilities in biochemical systems. The Journal of Chemical Physics

134(4), 01B628 (2011)

15. Dayar, T., Stewart, W.J.: Quasi lumpability, lower-bounding coupling matrices,

and nearly completely decomposable Markov chains. SIAM Journal on Matrix

Analysis and Applications 18(2), 482–498 (1997)

16. Gillespie, D.T.: Exact stochastic simulation of coupled chemical reactions. The

journal of physical chemistry 81(25), 2340–2361 (1977)

17. Golightly, A., Sherlock, C.: Eﬃcient sampling of conditioned Markov jump pro-

cesses. Statistics and Computing 29(5), 1149–1163 (2019)

18. Golightly, A., Wilkinson, D.J.: Bayesian inference for stochastic kinetic models

using a diﬀusion approximation. Biometrics 61(3), 781–788 (2005)

18 M. Backenk¨ohler et al.

19. Golightly, A., Wilkinson, D.J.: Bayesian parameter inference for stochastic bio-

chemical network models using particle Markov chain monte carlo. Interface focus

1(6), 807–820 (2011)

20. Grossmann, G., Backenk¨ohler, M., Wolf, V.: Importance of interaction structure

and stochasticity for epidemic spreading: A COVID-19 case study. In: Seventeenth

international conference on the quantitative evaluation of systems (QEST 2020).

IEEE (2020)

21. Ha jnal, M., Nouvian, M., ˇ

Safr´anek, D., Petrov, T.: Data-informed parameter syn-

thesis for population markov chains. In: International Workshop on Hybrid Systems

Biology. pp. 147–164. Springer (2019)

22. Hayden, R.A., Stefanek, A., Bradley, J.T.: Fluid computation of passage-time dis-

tributions in large Markov models. Theoretical Computer Science 413(1), 106–141

(2012)

23. He, S., Peng, Y., Sun, K.: SEIR modeling of the COVID-19 and its dynamics.

Nonlinear Dynamics pp. 1–14 (2020)

24. Henzinger, T.A., Mateescu, M., Wolf, V.: Sliding window abstraction for inﬁnite

Markov chains. In: International Conference on Computer Aided Veriﬁcation. pp.

337–352. Springer (2009)

25. Huang, L., Pauleve, L., Zechner, C., Unger, M., Hansen, A.S., Koeppl, H.: Re-

constructing dynamic molecular states from single-cell time series. Journal of The

Royal Society Interface 13(122), 20160533 (2016)

26. Kuwahara, H., Mura, I.: An eﬃcient and exact stochastic simulation method to an-

alyze rare events in biochemical systems. The Journal of chemical physics 129(16),

10B619 (2008)

27. Kwiatkowska, M., Norman, G., Parker, D.: Prism 4.0: Veriﬁcation of probabilistic

real-time systems. In: International conference on computer aided veriﬁcation. pp.

585–591. Springer (2011)

28. Lapin, M., Mikeev, L., Wolf, V.: SHAVE: stochastic hybrid analysis of Markov

population models. In: Proceedings of the 14th international conference on Hybrid

systems: computation and control. pp. 311–312 (2011)

29. Lipshtat, A., Loinger, A., Balaban, N.Q., Biham, O.: Genetic toggle switch without

cooperative binding. Physical review letters 96(18), 188101 (2006)

30. Loinger, A., Lipshtat, A., Balaban, N.Q., Biham, O.: Stochastic simulations of

genetic switch systems. Physical Review E 75(2), 021904 (2007)

31. Mikeev, L., Neuh¨außer, M.R., Spieler, D., Wolf, V.: On-the-ﬂy veriﬁcation and op-

timization of DTA-properties for large Markov chains. Formal Methods in System

Design 43(2), 313–337 (2013)

32. Mikeev, L., Sandmann, W.: Approximate numerical integration of the chemical

master equation for stochastic reaction networks. arXiv preprint arXiv:1907.10245

(2019)

33. Mikeev, L., Sandmann, W., Wolf, V.: Eﬃcient calculation of rare event proba-

bilities in Markovian queueing networks. In: Proceedings of the 5th International

ICST Conference on Performance Evaluation Methodologies and Tools. pp. 186–

196 (2011)

34. Mikeev, L., Sandmann, W., Wolf, V.: Numerical approximation of rare event prob-

abilities in biochemically reacting systems. In: International Conference on Com-

putational Methods in Systems Biology. pp. 5–18. Springer (2013)

35. Milner, P., Gillespie, C.S., Wilkinson, D.J.: Moment closure based parameter infer-

ence of stochastic kinetic models. Statistics and Computing 23(2), 287–295 (2013)

36. Mode, C.J., Sleeman, C.K.: Stochastic processes in epidemiology: HIV/AIDS, other

infectious diseases, and computers. World Scientiﬁc (2000)

Analysis of Markov Jump Processes under Terminal Constraints 19

37. Munsky, B., Khammash, M.: The ﬁnite state projection algorithm for the solution

of the chemical master equation. The Journal of chemical physics 124(4), 044104

(2006)

38. Neupane, T., Myers, C.J., Madsen, C., Zheng, H., Zhang, Z.: Stamina: Stochastic

approximate model-checker for inﬁnite-state analysis. In: International Conference

on Computer Aided Veriﬁcation. pp. 540–549. Springer (2019)

39. Pardoux, E.: Markov processes and applications: algorithms, networks, genome

and ﬁnance, vol. 796. John Wiley & Sons (2008)

40. Rabiner, L., Juang, B.: An introduction to hidden Markov models. IEEE ASSP

Magazine 3(1), 4–16 (1986)

41. S¨arkk¨a, S.: Bayesian ﬁltering and smoothing, vol. 3. Cambridge University Press

(2013)

42. Schnoerr, D., Cseke, B., Grima, R., Sanguinetti, G.: Eﬃcient low-order approxima-

tion of ﬁrst-passage time distributions. Phys. Rev. Lett. 119, 210601 (Nov 2017).

https://doi.org/10.1103/PhysRevLett.119.210601

43. Schnoerr, D., Sanguinetti, G., Grima, R.: Validity conditions for moment closure

approximations in stochastic chemical kinetics. The Journal of chemical physics

141(8), 08B616 1 (2014)

44. Siegal-Gaskins, D., Mejia-Guerra, M.K., Smith, G.D., Grotewold, E.: Emergence

of switch-like behavior in a large family of simple biochemical networks. PLoS

Comput Biol 7(5), e1002039 (2011)

45. Strasser, M., Theis, F.J., Marr, C.: Stability and multiattractor dynamics of a

toggle switch based on a two-stage model of stochastic gene expression. Biophysical

journal 102(1), 19–29 (2012)

46. Ullah, M., Wolkenhauer, O.: Stochastic approaches for systems biology. Springer

Science & Business Media (2011)

47. Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Courna-

peau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J.,

Brett, M., Wilson, J., Jarrod Millman, K., Mayorov, N., Nelson, A.R.J., Jones, E.,

Kern, R., Larson, E., Carey, C., Polat, ˙

I., Feng, Y., Moore, E.W., Vand erPlas, J.,

Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R.,

Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P., Contributors, S...:

SciPy 1.0: Fundamental Algorithms for Scientiﬁc Computing in Python. Nature

Methods 17, 261–272 (2020). https://doi.org/https://doi.org/10.1038/s41592-019-

0686-2

48. Wildner, C., Koeppl, H.: Moment-based variational inference for Markov jump

processes. arXiv preprint arXiv:1905.05451 (2019)

49. Wilkinson, D.J.: Stochastic modelling for systems biology. CRC press (2018)

50. Zapreev, I., Katoen, J.P.: Safe on-the-ﬂy steady-state detection for time-bounded

reachability. In: Third International Conference on the Quantitative Evaluation of

Systems-(QEST’06). pp. 301–310. IEEE (2006)