Optimal Control of a Fed-batch Fermentation Process by Neuro-Dynamic Programming

Abstract

In this paper the method for optimal control of a fermentation process is presented, that is based on an approach for optimal control - Neuro-Dynamic programming. For this aim the approximation neural network is developed and the decision of the optimization problem is improved by an iteration mode founded on the Bellman equation. With this approach computing time and procedure are decreased and quality of the biomass at the end of the process is increased.

Full text

Page 1

Bioautomation, 2004, vol. 1, pp. 57-66

57
Optimal Control of a Fed-batch Fermentation Process
by Neuro-Dynamic Programming

Tatiana Ilkova, Stoyan Tzonkov

Centre of Biomedical Engineering, Bulgarian Academy of Sciences
105, Acad. G. Bonchev Str., Sofia 1113, Bulgaria

E-mail: tanja@clbme.bas.bg
tzonkov@clbme.bas.bg


Abstract: In this paper the method for optimal control of a fermentation process is
presented, that is based on an approach for optimal control-Neuro-dynamic programming.
For this aim the approximation neural network is developed and the decision of the
optimization problem is improved by an iteration mode founded on the Bellman equation.
With this approach computing time and procedure are decreased and quality of the biomass
at the end of the process is increased.
Keywords: Fermentation process, Neuro-dynamic programming, Optimal control.


Introduction
The optimal control of the Fermentations Processes (FP) usually depends on the presence of
complex, non-linear dynamic model of the system because of this is difficult to realize to
working-out of the problem, which is very important to practical realize [7,9]. The problems
of the optimization difficulties arise even in offline optimal control, especially when a model
is a high dimensional, one with larger diapason of the investigations, lager composition of the
variables also of technological and optimizations’ restrictions [10].
One approach for solving to -problem of dynamic optimization is Dynamic Programming
(DP), and is successfully applied to fermentation processes [3,4,8]. However, the approach is
largely considered impractical as analytical solution of resulting dynamic program is seldom
possible and numerical solution suffers from the “curse of dimensionally” [3,4].

Neuro-Dynamic Programming (NDP) is suggested as a method to alleviation the “curse of
dimensionally”. The name neuro-dynamic programming expresses the reliance of the methods
of this article on both DP and neural network (NN) concepts [2]. The method is successfully

Page 2

Bioautomation, 2004, vol. 1, pp. 57-66

58
applied for optimal control of batch’s and continuous FP in the last years, as the computing
time was decreased about 2/3 and increases of the quantity of the desired products was gotten
[7,8].

In this paper an optimal control of a fed-batch FP process by NDP is presented and compared
it with the DP optimal control.

Formulation of the optimization problem
The problem of dynamic optimization includes minimization of the following expression [8]:
)(),( min
,...,
0
p
1p
∑
=
1i
ii
uu
xux
1p
φ+φ
−
−
(1), (1)
with constraints: 0 . 0),(
≥
iiuxg
;
10
−≤≤
pi
,
0)(
≥
p xg
;
),( uxfx =
?
,
for a given initial state x0 and a function constant input
hihiuiu
i
⋅+<τ≤⋅=
) 1()(, h is the
sampling time, xi-represent the value of x at the stage ith (т.е. x(t) в t=h.i), φ-single state cost
function and φ is the terminal state cost function and whey are defined on the multitude of
the real numbers, and f is continuous and differentiated function.
Bellman Equation

The DP involves stage-wise calculation of the cost-to-go function to arrive at the solution, not
just for a specific x0 but for general x0. For (1), the cost-to-go at each stage is defined as [3]:
)(),( min
,...,
i
−
p
1p
∑
=
ipj
jj
uu
i
xuxJ
1pp
φ+φ=
−
−
−
. (2)
and the next step:
)(),( min
1
i
−−
1
,...,
1
p
p
ipj
jj
uu
i
xuxJ
pp
φ+φ
−
=
∑
=−
+
.
If we mark )),((
uxFLJ
hi
⋅=
, where Fh(x,u) is the resulting state after integrating the
differential equation for one sample interval with the starting state of x and constant input of
u, L is continuous and differentiated function, then:
)),((),( min
1
i
−−
1
,...,
1
uxFLuxJ
h
p
ipj
jj
uu
i
pp
⋅+φ
−
=
∑
=−
+
(3)

Page 3

Bioautomation, 2004, vol. 1, pp. 57-66

59
In conformity with the “principle of the optimum” of Bellman [1] “The tail policy is optimal
for the tail subproblem”, at each stage the calculation of the cost-to-go function can be done
as:
)),((),(min
u
)(
1
uxFJuxxJ
hii
−
+φ=
, (4)
Equation (4) is calculated from i=1 to i=p
Such as, the pertinent terminal needs to be imposed at each stage. Once received, the cost-to-
go function has optimal decision for a general state x0 according to “principle of the
optimum” [1].

The objective of DP is to calculate numerically the optimal cost function J. This computation
can be done off-line, i.e., before the real system starts operating. In very few cases can we
solve the stage-wise optimization analytically to obtain a closed-form expression for the cost-
to-go function. An optimal policy, that is, an optimal choice of u for each i, is computed either
simultaneously with J, or in real time by minimizing in the right-hand side of Bellman’s
equation. It is well known, however, that for many important problems the computational
requirements of DP are overwhelming, mainly because of a very large number of states and
controls. In such situations a suboptimal solution is required.

Cost-to-go Approximation
The traditional approach to solving the Bellman equation involves gridding of the state space,
solving the optimization (4) for each grid point, and performing the stage-wise optimization
until convergence. Exhaustive sampling of state space can be avoided by identifying relevant
regions of the space through simulation under judiciously chosen suboptimal policies. The
policy improvement theorem states that a new policy that is greedy with respect to the cost-
to-go function of the original policy is as good as or better than the original policy (the now
policy is given when moment value of the cost-to-go function is least, i.e. the new policy is
defined by expression:
)),((),( min
u
arg)(
uxFJuxxu
hi
+φ=
),
where
),,(arg
ixuG
is arguments of the function∈ Rm+n+r, and u∈R m, x∈Rn and i∈Rr, and it is
improvement of the original policy. When the new policy is as good as the original policy, the
above equation becomes the same as Bellman optimality equation (4). When the iterations

Page 4

Bioautomation, 2004, vol. 1, pp. 57-66

60
converge, this off line-computed cost-to-go approximation optimal control calculation for the
bioreactor.

NDP Algorithm
NDP method is suboptimal methods that center around the approximate evaluation of the
optimal cost function J, possibly through the use of neural networks and/or simulation [2]. For
description of the algorithm the next symbol is used: J-represents cost-to-go values,
)(
~xJ
-a
function approximation relating J to corresponding state x, ()i iteration index for cost iteration
~
J
loop, k-discrete time. Then it can be written: ))((
~
J
)(
kxk
≡
.
If the system starts with a given policy (some rule for choosing a decision u at each possible
state i), and its approximately evaluate the cost of that policy (as a function of the current
state) by least-squares-fitting a scoring function )(
~kJ
to the results of many simulated system
trajectories using that policy and simulation of the process with chosen suboptimal policies
under all representative operating conditions. The cost-to-go function is calculated using the
simulation data for each state visited during the simulation, as for each closed loop simulation
yields us data x(0), x(1), …, x(N), where N is sufficiently large for the system to reach
equilibrium, calculated of one-stage cost φ(k) for these points. Then cost-to-go is the sum of
single stage costs from the next point -
∑
=
k
+
φ
1
=
N
i
ikJ
)()(. This is step simulation. After we
preparing of а NN to the data approximate the cost-to-go function – denoted as )(
~0xJ
, as a
smooth function of the states. The NN is chosen with dependent on number of system
equation, Bellman’s equation and time. A new policy is then defined by minimization in
Bellman’s equation, where the optimal cost is replaced by the calculated scoring function, and
the process is repeated. To improve the approximation, perform the following iteration
(referred to as the cost iteration) until convergence:
• With the current cost-to-go approximation
)(
~
Ji
x
is calculated
)(
~
Ji+
1k
for the given
points of x by following equation:
~
),(min
Jux
u
)),((
~
J
1
uxF
h
ii
+φ=
+
, (5)
which is based the Bellman equation. This step is approximation.
Update of the policy. It may necessary to increase the coverage of the state space i.e. more
suboptimal simulations with the updated policy with suboptimal policy for improvement of

Page 5

Bioautomation, 2004, vol. 1, pp. 57-66

61
cost-to-go approximation are used to increase the coverage or the number of data points in
certain region of state space.

Application of the NDP for optimal control of a fermentation process
The process of a fed-batch fermentation of E. Coli is examined in this work. The fermentation
is leaded in a bioreactor with mixing in mesophyle regime. The parameters of the process are
showed in Table 1 [6, 7].
Table 1 Parameters of the process
Parameter Value
Temperature 35°C
pH 6,8
Gassing rate 275 1/h
Oxygen around 35 %
Glucose in Batch 2,5 g/L
Volume 1,5 g/L
Glucose in Feeding Solution 100 g/L
Stirrer speed at start 900 rpm
Stirrer speed at end 1500 rpm


The Model of the Process
The program was made for parametric identification and estimate of the parameter of the
model on the basis of non-linear regression. On the basis of this program the following kinetic
model was received [5]:
V
F
X
dt
dX
−µ= . (6)
()
XSS
V
F
dt
dS
0
. η−−=
(7)
F
dt
dV=
, (8)
where:
Sk
S
S+
⋅µ=µ
max
;
m
k
Y+=µ
η
;