# ANNSA: a hybrid artificial neural network/simulated annealing algorithm for optimal control problems

**ABSTRACT** This paper introduces a numerical technique for solving nonlinear optimal control problems. The universal function approximation capability of a three-layer feedforward neural network has been combined with a simulated annealing algorithm to develop a simple yet efficient hybrid optimisation algorithm to determine optimal control profiles. The applicability of the technique is illustrated by solving various optimal control problems including multivariable nonlinear problems and free final time problems. Results obtained for the different case studies considered agree well with those reported in the literature.

**0**Bookmarks

**·**

**71**Views

- [Show abstract] [Hide abstract]

**ABSTRACT:**The solutions of dynamic optimization problems are usually very difficult due to their highly nonlinear and multidimensional nature. Genetic algorithm (GA) has been proved to be a feasible method when the gradient is difficult to calculate. Its advantage is that the control profiles at all time stages are optimized simultaneously, but its convergence is very slow in the later period of evolution and it is easily trapped in the local optimum. In this study, a hybrid improved genetic algorithm (HIGA) for solving dynamic optimization problems is proposed to overcome these defects. Simplex method (SM) is used to perform the local search in the neighborhood of the optimal solution. By using SM, the ideal searching direction of global optimal solution could be found as soon as possible and the convergence speed of the algorithm is improved. The hybrid algorithm presents some improvements, such as protecting the best individual, accepting immigrations, as well as employing adaptive crossover and Gaussian mutation operators. The efficiency of the proposed algorithm is demonstrated by solving several dynamic optimization problems. At last, HIGA is applied to the optimal production of secreted protein in a fed batch reactor and the optimal feed-rate found by HIGA is effective and relatively stable.Chinese Journal of Chemical Engineering 02/2013; 21(2):144–154. · 0.87 Impact Factor - [Show abstract] [Hide abstract]

**ABSTRACT:**An approach that combines genetic algorithm (GA) and control vector parameterization (CVP) is proposed to solve the dynamic optimization problems of chemical processes using numerical methods. In the new CVP method, control variables are approximated with polynomials based on state variables and time in the entire time interval. The iterative method, which reduces redundant expense and improves computing efficiency, is used with GA to reduce the width of the search region. Constrained dynamic optimization problems are even more difficult. A new method that embeds the information of infeasible chromosomes into the evaluation function is introduced in this study to solve dynamic optimization problems with or without constraint. The results demonstrated the feasibility and robustness of the proposed methods. The proposed algorithm can be regarded as a useful optimization tool, especially when gradient information is not available.Engineering Optimization 09/2013; 45(9). · 1.23 Impact Factor - [Show abstract] [Hide abstract]

**ABSTRACT:**Two general approaches are adopted in solving dynamic optimization problems in chemical processes, namely, the analytical and numerical methods. The numerical method, which is based on heuristic algorithms, has been widely used. An approach that combines differential evolution (DE) algorithm and control vector parameterization (CVP) is proposed in this paper. In the proposed CVP, control variables are approximated with polynomials based on state variables and time in the entire time interval. Region reduction strategy is used in DE to reduce the width of the search region, which improves the computing efficiency. The results of the case studies demonstrate the feasibility and efficiency of the proposed methods.Chinese Journal of Chemical Engineering 01/2013; 21(1):64–71. · 0.87 Impact Factor

Page 1

Chemical Engineering Science 58 (2003) 3131–3142

www.elsevier.com/locate/ces

ANNSA: a hybrid arti?cial neural network/simulated annealing

algorithm for optimal control problems

Debasis Sarkar, Jayant M. Modak∗

Department of Chemical Engineering, Indian Institute of Science, Bangalore 560012, India

Received 15 July 2002; received in revised form 12 November 2002; accepted 26 March 2003

Abstract

This paper introduces a numerical technique for solving nonlinear optimal control problems. The universal function approximation

capability of a three-layer feedforward neural network has been combined with a simulated annealing algorithm to develop a simple yet

e?cient hybrid optimisation algorithm to determine optimal control pro?les. The applicability of the technique is illustrated by solving

various optimal control problems including multivariable nonlinear problems and free ?nal time problems. Results obtained for the di?erent

case studies considered agree well with those reported in the literature.

? 2003 Elsevier Ltd. All rights reserved.

Keywords: Neural networks; Simulated annealing; Optimisation; Control; Bioreactors; Chemical reactors

1. Introduction

Optimal control problems appear frequently in many dif-

ferent industries and the determination of optimal control

trajectories is extremely important for better performance of

process industries. Typical examples in chemical engineer-

ing include temperature pro?le determination for batch re-

actors, substrate feed rate pro?les for fed-batch bioreactors,

and catalyst distribution in packed bed reactors. Since the

objective is to determine the trajectories of the optimal con-

trol variables as a function of time or length of the reactor,

these problems fall under the realm of variational calculus.

In general, the solution to such problems is often a di?cult

and time-consuming task even for a single control variable

problem. There are several methods available to solve opti-

mal control problems and they can be classi?ed broadly into

two major categories.

One class of optimisation methods uses variational

approach based on the optimal control theory and thus re-

quires the solution of multipoint boundary value problems

with initial conditions for state variables and terminal con-

ditions for adjoint (costate) variables. Analytical solutions

can be determined using the optimality conditions only

when the number of state variables is few and the numerical

∗Corresponding author. Tel.: +91-80-3942768; fax: +91-80-3608121.

E-mail address: modak@chemeng.iisc.ernet.in (J. M. Modak).

solutions are usually unavoidable. Several numerical tech-

niques using the optimality conditions have been reported in

the literature. These include among others multiple shooting

methods for multipoint boundary value problems (Oberle

& Sothmann, 1999), boundary condition iterative methods

(Ray, 1981), and gradient methods (Stutts, 1983). These are

the most elegant and precise methods to solve optimal con-

trol problems and the characteristic properties of the opti-

mal solutions can be clearly identi?ed here. However, these

methods require very good initial guesses for either con-

trol variable trajectory or unknown boundary conditions in

order to achieve convergence. Additionally, the switching

structure (sequence of maximum, minimum, and interme-

diate control) has to be guessed correctly in advance in

some cases. The other class of optimisation methods dis-

cretises the independent variable (time for batch/fed-batch

reactors or axial position for packed bed reactors) horizon

into ?nite subintervals and piece-wise constant or linear

control functions in the discretized intervals are determined

using di?erent search techniques (Shukla & Pushpavanam,

1998; Banga, Irrizarry-Rivera, & Seider, 1998; Roubos, van

Straten,&vanBoxtel,1999;Luus,2000;Na,Chang,Chung,

& Lim, 2002). Usually, these techniques are more robust in

terms of convergence. Moreover, the switching structure of

the optimal solution need not be guessed at all.

An alternate strategy to avoid discretisation can be the

representation of the control variable pro?le as a single

0009-2509/03/$-see front matter ? 2003 Elsevier Ltd. All rights reserved.

doi:10.1016/S0009-2509(03)00168-4

Page 2

3132D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

nonlinear function of the independent variable. However,

such a function must be capable of capturing the character-

istic nonlinearities associated with the optimal control prob-

lems. For example, in the case of singular control problems,

so called because control variable appears linearly in the

state and/or performance index, the control variable trajec-

tory will have bang-bang and/or singular arcs. The con-

tinuous function chosen to represent the optimal control

pro?le must be capable of displaying the sharp discontinu-

ities. However, constructing such a nonlinear function may

be di?cult in practice.

There has been considerable interest recently in the vari-

ous possibilities o?ered by Neural Networks and they have

been successfully applied for various interesting applica-

tions (Baughman & Liu, 1995). A key feature of neural net-

works is their ability to approximate any arbitrary nonlinear

function and this provides an excellent means of generating

a continuous control pro?le in combination with any global

optimisation technique. Simulated annealing (SA), a prob-

abilistic optimisation technique with a potential of ?nding

global optimal solution, is based on the ideas from statistical

mechanics(Kirkpatrick,Gelatt,&Vecchi,1983).Inthe?eld

of chemical engineering, SA has been successfully applied

in various problems like optimisation of heat exchanger

networks (Dolan, Cummings, & Le Van, 1989), separation

sequence synthesis (Floquet, Pibouleau, & Domenech,

1994), optimisation of reactive distillation processes

(Cardoso, Salcedo, de Azevedo, & Barbosa, 2000), optimi-

sation of batch distillation processes (Hanke & Li, 2000),

biotechnical processes (Simutis & Lubbert, 1997), etc.

The purpose of the present study is to exploit nonlinear

function approximation capability of a neural network to

develop a simple yet e?cient hybrid optimisation strategy

by combining a multilayer feedforward neural network with

a Simulated Annealing algorithm. The e?ectiveness and the

?exibility of the proposed hybrid Arti?cial Neural Network

Simulated Annealing (ANNSA) approach is demonstrated

with four di?erent optimal control problems taken from the

literature.

2. Optimisation problem formulation

Letusconsiderasystemdescribedbythefollowingvector

di?erential equation:

dx

dt= f(x;u);(1)

with the initial conditions speci?ed as x(0). The state vector

x is an (n × 1) vector and u is an (m × 1) control vector

bounded by

umin6ui6umax;i = 1;2;:::;m: (2)

The objective is to determine the optimal control policy u(t)

in the time interval 06t 6tfwhich will maximise a scalar

index of performance (PI) that depends on the ?nal outcome

of the process:

Maximise

u(t)

PI = ?(x(tf));(3)

where tfis the ?nal time that may be speci?ed a priori or

free.

3. Neural network for the time/control relationship

In the present study, the role of a neural network is to ap-

proximate the control variable which is a continuous func-

tion of time. A neural network with a single hidden layer

has been proved to approximate any arbitrary function with

any arbitrary precision, provided there are su?cient num-

ber of neurons in the hidden layer (Cybenko, 1989; Hornik,

Stinchcombe, & White, 1989). This ability of arti?cial neu-

ral networks (ANN) to approximate any arbitrary nonlinear

functions can be exploited to determine the control function

of time in optimal control problems.

Since neural network computations are usually performed

by scaling input and output quantities in the range of 0–1,

let us consider a scaled control vector ˆ u(?) where ? is the

scaledtime.Accordingtothefunctionapproximationtheory,

there should be an approximating function F(?;P) having

a ?xed number of real-valued parameters P∈R which will

e?ectively approximate the function ˆ u(?):

ˆ u(?) = F(?;P);(4)

where F is the ANN model and P represents the vector of

tunable parameters of the network, namely, its weights and

the biases. Thus, for a choice of a speci?c F (i.e. ANN),

the problem is to ?nd the set of parameters P that provides

the best possible approximation of ˆ u(?). The problem of de-

termining the optimal control pro?le, therefore, reduces to

a nonlinear programming problem where the decision vari-

ables are the appropriate weights and biases of the chosen

neural network:

Maximise

P

and it can, in general, be solved using any nonlinear

optimisation technique. In the present study, we choose sim-

ulated annealing for this purpose because of its simplicity

and ability to produce global optimal solutions for complex

problems which outweigh its relatively large computational

requirements.

One of the most important tasks in using neural networks

is to decide on the architecture of the network. For the prob-

lems we study here, we chose a three-layer feedforward

network (Fig. 1) with one neuron in the input layer corres-

ponding to the time, a single hidden layer of NH neurons,

and an output layer of m neurons corresponding to the num-

ber of control variables. A simple sigmoidal function is used

as an activation function for all the neurons in hidden and

output layers. We refer to this network as 1−NH−m ANN.

PI = ?(x(tf))(5)

Page 3

D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142 3133

θj

b

bj

τ

Bias unit

Hidden layer

Bias unit

Input layer

Wij

Output layer

out i,

(m)

(1)

u^ u^

1m

( NH)

Fig. 1. A three-layer feedforward ANN architecture.

With this network, the control vector ˆ u = [ ˆ u1ˆ u2::: ˆ um]Tis

represented as follows:

1

1 + e−neti(?);

where neti(?) is

NH

?

j=1

ˆ ui(?) =i = 1;2;:::;m; (6)

neti(?) =Wij’(?j? + bj) + bi;out:(7)

Wijis the weight connecting jth hidden layer neuron to the

ithoutputneuron,?jistheweightconnectinginputneuronto

the jth hidden layer neuron, bjis the bias for the jth hidden

layer neuron, bi;outis the bias for the ith output neuron, and

’(?j?+bj) is the output of the jth hidden layer neuron upon

application of sigmoidal activation function:

1

1 + e−(?j?+bj);

For a 1−NH−m ANN used in this study, there are 2NHcon-

nection parameters (?j;bj) between input and hidden layers,

and m(NH+ 1) connection parameters (Wij;bi;out) between

hidden and output layers. Thus, the total number of param-

eters in set P = {Wij;?j;bj} is N = 2NH+ m(NH+ 1).

One of the key features of an optimal control pro?le,

particularly for singular control problems, is the sharp jumps

between maximum, minimum, and intermediate values for

the control variable. As we use an ANN to approximate the

control pro?le, the ANN must be able to predict these sharp

edges when the control variable switches from one level to

’(?j? + bj) =j = 1;:::;NH:(8)

other. Since ANN is a universal function approximator, it

can indeed predict these sharp changes in the pro?le, pro-

vided there are enough neurons in the hidden layer as dis-

cussed below.

Let us consider a typical bang-bang policy ˆ u1(?) with ns

switches as shown in Fig. 2A for ns= 3. To approximate

this bang-bang policy with nsswitches, ˆ u1(?) must have the

same value at nsand di?erent values of ? as shown by the

dotted line in Fig. 2A. In view of the unimodal sigmoidal

activation function used (Eq. (6)), the same value of net1(?)

for the nsdi?erent values of ? must be predicted by the ANN

model. In other words,

net1(?) = c (9)

must have nsroots, where c is any arbitrary constant. For

the 1−NH−1 ANN model described above (Eqs. (6)–(8)),

Eq. (9) can be written as

W11

1 + ?1

1 + ?2

1 + ?NH

where ?jis e−(?j?+bj). Eq. (10) can be simpli?ed and ex-

pressed as a sum of exponential functions in the following

form:

+

W12

+ ··· +

W1NH

= c;(10)

c1e−?1?+ c2e−?2?+ ··· + cle−?l?= 0;

where ciand ?i, i = 1;2;:::;l are real constants de?ned in

terms of parameter set P, and l=2NH. It is known that a sum

of l real exponential functions, with real coe?cients, has at

most (l−1) real zeros, where equal zeros are counted with

full multiplicity (Pontryagin, Boltyanskii, Gamkrelidze, &

Mischenks, 1962). Thus, Eq. (11) has at most (l − 1), i.e.

2NH− 1 real roots. This leads to the conclusion that to ap-

proximate a bang-bang policy with nsswitches, the follow-

ing must hold true:

(11)

2NH− 1¿ns:

It may be pointed out that Eq. (12) gives us the bare min-

imum number of hidden neurons that is necessary and for

a reasonably good approximation of the control policy we

may need more number of neurons in the hidden layer than

that predicted by the equality sign of the above Eq. (12).

Fig. 2b–d show the results of approximation of the

bang-bang policy given by Fig. 2A by a 1 − NH− 1 ANN

trained by backpropagation algorithm with various number

of hidden neurons (NH). In view of Eq. (12), the minimum

number of neurons necessary in the hidden layer is 2. It can

be seen from the ?gures that NH= 1 can predict only one

switch, and therefore fails to approximate the policy (Fig.

2b). NH = 2 satis?es Eq. (12) and predicts the required

number of switches. As a result, there is a qualitative match

between the predicted and desired policy (Fig. 2c), but the

quantitative agreement is not very good. It can be expected

that the introduction of additional neurons in the hidden

layer will improve the prediction capability of the ANN

model. This can be clearly seen in Fig. 2d which shows the

prediction of ANN model with NH= 3.

(12)

Page 4

3134 D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

0 0.10.2 0.30.4 0.5

Time ?

0.60.7 0.80.91

0

0.2

0.4

0.6

0.8

1

0 0.10.20.30.4 0.5

Time ?

0.60.7 0.80.91

0

0.2

0.4

0.6

0.8

1

0 0.10.2 0.3 0.40.5

Time ?

0.6 0.7 0.80.91

0

0.2

0.4

0.6

0.8

1

0 0.1 0.20.3 0.40.5

Time ?

0.6 0.7 0.80.91

0

0.2

0.4

0.6

0.8

1

(a)

(b)

(c)

(d)

Fig. 2. (a) Bang-bang control policy with three switches. (b–d) Prediction of control policy in (a) with various architectures of ANN. (b) 1-1-1, (c)

1-2-1, and (d) 1-3-1.

4. Implementation details of the proposed hybrid

algorithm

The SA part of our hybrid optimisation algorithm is

similar to the implementation of simulated annealing by

Go?e, Ferrier, and Rogers (1994). The details of our hybrid

optimisation algorithm ANNSA are outlined as follows:

Step 1: The SA algorithm starts at some speci?ed “high”

temperature T0with a single trial solution vector P (weights

and biases of chosen ANN) of dimension N. The elements

of P are initially generated randomly within a suitable upper

and a lower limit. The objective function (PI) for the opti-

mal control problem in Eq. (5) is now evaluated by solving

the di?erential equations describing the process (Eq. (1))

with the control pro?le u(t) generated by the chosen ANN

with the above weights and biases (P) in the speci?ed time

interval 06t 6tf. The IMSL subroutine DGEAR, which

is based on variable step size Gear’s method for the solv-

ing sti? di?erential equations (Gear, 1971), is used for the

solution of the di?erential equations. A maximum time step

of 0:01 h is used to obtain numerical solutions.

Step 2: A new candidate solution P?is now generated by

means of random perturbation of the current solution,

P?

i= Pi+ r?i;(13)

where r ∈[ − 1;1] is a uniformly distributed random num-

ber and ?i is the ith element of the step length vector ?

of dimension N. The new objective function value PI?

is then evaluated for this new solution as discussed in

Step 1.

Step 3: If PI?¿PI, the new solution P?is always ac-

cepted, P is set to P?and the algorithm moves uphill. But if

PI?6PI, the new solution P?is accepted with a probability

Page 5

D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–31423135

Table 1

SA parameters used for optimisation

Initial temperature, T0

Temperature reduction factor, ?

Feasible solution space

Step length vector, ?

NS

NT

NC

?

1.0

0.95

−100 to 100

100

20

20

4

10−6

of (Metropolis criterion)

P = exp(−?PI=T);

where ?PI=PI?−PI. If the new solution P?is accepted, P

is set to P?, and the algorithm moves downhill. Otherwise,

P?is rejected.

Step 4: After NS×N function evaluations, the step length

vector ? is so adjusted that 50% of all moves are accepted.

In order to simulate the thermal equilibrium at every temper-

ature, the temperature is reduced after NT times the above

loop (NT× NS× N function evaluations) according to the

following decrement function:

(14)

T?= ?T; (15)

where ? is a constant temperature reduction factor somewhat

less than unity.

Step 5: The algorithm ends by comparing the last NC

values of the largest function values from the end of each

temperature reduction with the most recent one and the op-

timum function value. If all these di?erences are less than a

pre-speci?ed small value ?, the algorithm terminates.

Table 1 summarises the values of the SA parameters that

have been used in this study unless mentioned otherwise. It

may be pointed out here that not much e?ort has been made

in ?ne tuning these parameters for better performance of the

algorithm.

5. Optimisation results and discussion

5.1. Case study I: Optimal production of secreted protein

in a fed-batch bioreactor

This optimal control problem deals with the maximisation

of the production of secreted heterologous protein by a yeast

strain in a fed-batch bioreactor with the feed ?ow rate as the

onlycontrolvariable.ThiswasoriginallyformulatedbyPark

and Ramirez (1988) and the required details of the model

can be found in their work. The objective is to maximise the

total secreted protein in the bioreactor at the end of fed-batch

operation for a speci?ed ?nal time, tf (=15 h). The feed

rate is constrained as 06u610 l=h.

SA alone can be used to solve this optimal control prob-

lem if the control function is approximated by a piece-wise

constant or piece-wise linear functions. Fig. 3 shows the

0510 15

0

0.5

1

1.5

2

2.5

3

Time (h)

Feed Rate (L/h)

Fig. 3. Optimal feed rate pro?le for secreted protein (Case Study I):

Simulated annealing with piece-wise constant feed.

optimal feed rate pro?le obtained for this protein secre-

tion process when the time horizon is divided into 50 equal

intervals and a piece-wise constant feed is used in each

interval. Although the pro?le yields a good performance

index (PI = 32:759055), it can be seen from the ?gure that

the pro?le shows high activity with lots of abrupt jumps

in the values of feed rates. This behaviour is characteristic

of many algorithms which approximate the control function

by piece-wise constant functions and some ?ne tuning of

the algorithm such as proper choice of number of intervals,

introduction of ?lters etc. is required to generate smoother

pro?les (Tholuder & Ramirez, 1997).

The proposed ANNSA algorithm was run with various

architectures of ANN and the results of the optimisation

studies are presented in Table 2. The pro?les for the control

variable (substrate feed rate) obtained for the various cases

are shown in Fig. 4a–e and Fig. 4f shows the optimal sub-

strate feeding policy that is obtained through the application

of optimal control theory (Park & Ramirez, 1988). Even

with as few hidden neurons as 3 (Fig. 4a), the predicted feed

rate policy displays all the characteristics of the optimal pol-

icy except the presence of the batch period in the time inter-

val 9.65–10:70 h (Fig. 4f). With ?ve neurons in the hidden

layer, ANNSA prediction of the feed pro?le agrees quali-

tatively with the optimal solution, that is, feeding sequence

is correctly predicted as singular, batch, singular, and max-

imum feed rates. The insensitivity of the control pro?le on

the yield (PI) is clearly seen from Table 2 as even a subop-

timal pro?le also yields a high performance index. The best

performance index obtained is 32.633647 (case E in Fig. 4)

and this is in good agreement with the values reported in the

literature for this bioreactor. Fig. 5 shows the convergence

pro?le for the case where we use a 1-6-1 ANN (Fig. 4e).

Page 6

3136 D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

Table 2

Optimal production of secreted protein

Case ANN

model

No. of decision

variables

Performance

index

Nevala

Final

temperature

×103

0.0045

0.2461

0.0017

0.2221

0.0232

A

B

C

D

E

1-2-1

1-3-1

1-4-1

1-5-1

1-6-1

7 30.894534

32.359190

32.414095b

32.630342

32.639583

674801

652001

6734001

1056001

1588401

10

13

16

19

aNumber of function evaluations.

bSA parameters: NT=100, feasible solution space: −200 to +200, ? = 200.

05 1015

0

2

4

6

8

10

Time (h)

Feed Rate (L/h)

05 1015

0

2

4

6

8

10

Time (h)

Feed Rate (L/h)

05 10 15

0

2

4

6

8

10

Time (h)

Feed Rate (L/h)

05 10 15

0

2

4

6

8

10

Time (h)

Feed Rate (L/h)

05 1015

2

4

6

8

10

Time (h)

Feed Rate (L/h)

05 1015

0

2

4

6

8

10

Time (h)

Feed Rate (L/h)

(a)

(b)

(c)

(d)

(e) (f)

Fig. 4. Optimal feed rate pro?les for secreted protein (Case Study I): ANNSA with various architectures of ANN: (a) 1-2-1, (b) 1-3-1, (c) 1-4-1, (d)

1-5-1, (e) 1-6-1, and (f) feed pro?le using optimal control theory.

Page 7

D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–31423137

100

101

102

103

104

105

106

0

5

10

15

20

25

30

35

Number of function evaluations

Performance index

Fig. 5. Convergence pro?le of ANNSA algorithm for secreted protein

problem (Case Study I) with 1-6-1 ANN.

The nonlinearity of the optimal feed rate policy (Fig. 4f)

results from: (i) the time varying feed rate rates during the

singular intervals and (ii) the sequence of feed rate among

maximum, minimum, and singular interval feed rate with

sudden switching among them. It is instructive to examine

as to how an ANN model predicts these two aspects of time

varying feed rate policy. This is achieved by examining the

variation of output of each neuron in the hidden layer (Eq.

(8)) with time. Fig. 6 shows the output of each neurons

in the hidden layer for the 1-6-1 ANN (case E in Fig. 4).

Because of the sigmoidal activation function, the output of

theseneuronsareboundedbetween0and1.Acertaindegree

of order can be seen from the contributions of the individ-

ual neurons. It is interesting to note that not all the neurons

are “active” (that is, nonzero contribution) throughout the

entire operating period of 15 h. Furthermore, most of the

neurons remain either “on” with unity output or “o?” with

zero output. For example, the 1st neuron has a constant con-

tribution of unity till 9:6 h followed by rapidly decreasing

contribution till 10:8 h and no contribution thereafter. The

time interval of 9.6–10:8 h corresponds to the switching of

feed rate from 9.65–10:70 h (Fig. 4f). In other words, the

sudden switches in the optimal feed rate is emulated by the

on/o? nature of the neurons. Except for the 4th neuron, all

the neurons in the hidden layer showed this type of be-

haviour. The output of the 4th neuron varies continually

throughout the operating period and this neuron is mainly

responsible for the prediction of feed rate variation with time

during the singular intervals.

5.2. Case study II: induced foreign protein production by

recombinant bacteria

Lee and Ramirez (1994) have developed the optimal

nutrient and inducer feeding strategy for the fed-batch

production of induced foreign protein using recombinant

bacteria. They used the optimal control theory and showed

the existence of singular control arcs for this system. There-

after, this fed-batch system with two control variables has

been widely used for optimal control studies by various

algorithms (Carrsco & Banga, 1997; Roubos et al., 1999;

Mekarapiruk & Luus, 2000; Jayaraman, Kulkarni, Gupta,

Rajesh, & Kusumaker, 2001). The details of the model can

be found in Lee and Ramirez (1994). The objective is to

maximise the pro?tability of the process for a speci?ed

?nal time of fed-batch operation. This can be described

mathematically by the following performance functional:

Maximise

u(t)

PI = x4(tf)x1(tf) − Q

?tf

t0

u2(t)dt; (16)

wherex4istheforeignproteinconcentration,x1isthereactor

volume, and Q is the ratio of the cost of the inducer to the

value of the protein product. The fermentation time is ?xed

at 10 h. The nutrient (glucose) feed rate (u1) and the inducer

feed rate (u2) values are constrained as 06ui61:0, i=1;2.

Two di?erent cases have been considered for this

two-control variable problem: (i) the cost of the inducer

can be neglected, Q = 0, and (ii) the cost of the inducer

cannot be neglected, Q = 5. Three di?erent ANN architec-

tures (1-3-2, 1-4-2, and 1-5-2) were used for this problem

and the results of our optimisation studies are presented in

Table 3. Fig. 7 shows the optimal control pro?les that are

obtained with a 1-4-2 ANN for both the cases when Q = 0

and 5. It can be seen that when the cost of the inducer can

be neglected (Q = 0) both the feed rates have to be varied

for maximum performance, whereas when the cost of the

inducer cannot be neglected (Q = 5) only the inducer feed

rate has to be varied, as the nutrient feed rate is maintained

at zero for the entire period of operation. Table 4 shows the

performance index reported in the literature for this system

using di?erent methods. It can be seen that the results of

our ANNSA algorithm match favourably with the reported

results.

5.3. Case study III: isothermal CSTR with complex

reactions

This problem considers a nonlinear system consisting of

?ve simultaneous reactions taking place in an isothermal

CSTR. There are four control variables and thus it serves

well to test the e?ciency of the proposed algorithm. This

optimal control problem has been studied by Luus (1990)

to test the e?ciency of the dynamic programming algorithm

and the required details of the model can be found in the

above reference. The objective is to maximise the economic

bene?t of the CSTR by choosing u1;u2;u3, and u4in the

Page 8

3138 D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

05 10 15

0

0.2

0.4

0.6

0.8

1

Time (h)

1st Neuron

05 1015

0

0.2

0.4

0.6

0.8

1

Time (h)

2nd Neuron

05 10 15

0

0.2

0.4

0.6

0.8

1

Time (h)

3rd Neuron

05 10 15

0

0.2

0.4

0.6

0.8

1

Time (h)

4th Neuron

05 1015

0

0.2

0.4

0.6

0.8

1

Time (h)

5th Neuron

05 10 15

0

0.2

0.4

0.6

0.8

1

Time (h)

6th Neuron

Fig. 6. Contribution of each hidden neuron for 1-6-1 ANN for feed rate in Fig. 4e.

Table 3

Induced foreign protein production

CaseANN

model

No. of decision

variables

Q Performance

index

Nevala

Final

temperature

×105

0.0748

0.0448

0.0404

A

B

C

1-3-2

1-4-2

1-5-2

14

18

22

0

0

0

1.009750

1.009796

1.009783

1545601

2059201

2534401

D

E

F

1-3-2

1-4-2

1-5-2

14

18

22

5

5

5

0.816620

0.816699

0.816713

1338401

2023201

2120801

0.4991

0.0578

0.4504

aNumber of function evaluations.

Page 9

D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–31423139

02468 10

0

0.2

0.4

0.6

0.8

1

1.2

Time (h)

u1 (L/h)

02468 10

0

0.2

0.4

0.6

0.8

1

1.2

Time (h)

u2 (L/h)

024

Time (h)

68 10

0

0.2

0.4

0.6

0.8

1

1.2

u1 (L/h)

024

Time (h)

68 10

0

2

4

6

8

10

12x 10

3

u2 (L/h)

(a)(b)

(c) (d)

Fig. 7. Optimal feed rate (u1and u2) pro?les for induced foreign protein production (Case Study II): ANNSA with 1-4-2 ANN. (a), (b): Q = 0; (c),

(d): Q = 5.

time interval 06t 6tf. The ?nal time tf is speci?ed as

0:2 h a priori. The constraints on the control variables are:

06u1620, 06u266, 06u364, and 06u4620.

Luus (1990) reported the results of two sets of optimisa-

tion studies for this problem. In the ?rst case, u4is kept ?xed

at 6.0 and the optimisation is carried out on u1;u2, and u3.

Rao and Luus (1972) also solved this three-control problem

with an improved control vector iteration procedure. In the

second case, all the four control variables are taken free. In

order to compare our results with those reported in the litera-

ture, we ?rst solved this problem by keeping u4 ?xed at

6.0. The ANN used to determine the optimal trajectories for

u1;u2, and u3is 1-3-1. Fig. 8

pro?les obtained for the control variables u1;u2; and u3. This

results in a performance index of 20.083093 which agrees

well with that reported by Luus (1990) using dynamic pro-

gramming (20.10 with 40 stages). The control pro?les ob-

tained from our ANNSA are also in excellent agreement,

both qualitatively and quantitatively, with the pro?les ob-

tained by Rao and Luus (1972) using control vector iteration

who reported a maximum performance index of 20.09. Next,

the problem is solved by the ANNSA algorithm for the case

when all the control variables are free. Table 5 summarises

the results of our optimisation study for various architec-

presents the optimal control

Table 4

Induced foreign protein production: a comparative study

Method Performance index

Q = 0Q = 5

Optimal control theory

(Lee & Ramirez, 1994)

Adaptive stochastic algorithm

(Carrsco & Banga, 1997)

Evolution strategy

(Roubos et al., 1999)

Iterative dynamic programming

(Mekarapiruk & Luus, 2000)

Ant colony algorithm

(Jayaraman et al., 2001)

Proposed ANNSA with 1-4-2 ANN

1.0012 0.7988

1.00970.8165

Not reported0.8149

1.0096040.816480

1.00194 0.8095

1.0097960.816699

tures of ANN. Fig. 9 presents the optimal control pro?les

for all the four variables for the 1-5-4 ANN. The pro?les are

in good agreement with those reported by Luus (1990). The

performance indices obtained in this study (21.78, Table 5)

also match well with the value reported by Luus (21.76 with

11 stages). The optimal control policy for four control prob-

lems is completely di?erent from the case with three control

Page 10

3140 D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

0 0.050.1 0.150.2

0

2

4

6

8

10

12

14

16

18

20

Control

Time (h)

u1

u2

u3

Fig. 8. Optimal control pro?les for isothermal CSTR with three control

variables (Case Study III): ANNSA with 1-3-3 ANN.

variables. Also, the total ?ow rate (u1+ u2+ u4) is zero

in the time interval 0:0926t 60:122 and this makes the

CSTR semicontinuous under the optimal operating condi-

tions. A similar observation has also been reported by Luus

(1990). It may be pointed out that the gain in performance

index by allowing u4to be a free variable is about 8%.

5.4. Case study IV: biphasic growth of yeast

Pyun, Modak, Chang, and Lim (1989) reported the opti-

misation of fed-batch growth of yeast for various initial con-

ditions of the bioreactor using optimal control theory. We

consider here case-A from their study where the initial sub-

strate (glucose) and cell concentrations are low. The optimal

control problem is to determine the optimal glucose feed-

ing policy that maximises the pro?t de?ned as the di?erence

Table 5

Isothermal CSTR with complex reactions

CaseANN

model

No. of decision

variables

Performance

index

Nevala

Final

temperature

×104

0.0188

0.3689

0.0582

A

B

C

1-3-4

1-4-4

1-5-4

22

28

34

21.160195

21.785739

21.789844

2270401

2240001

8024001

aNumber of function evaluations.

between the product value (cell mass) and the operating

cost:

Maximise

u(t)

PI = (XV)tf− ?tf; (17)

where X is the concentration of cells and V is the fermentor

volume. The ? is a cost factor, a composite operating cost

per unit time per unit yeast cell mass price. The ?nal time tf

is not speci?ed a priori and has to be determined optimally.

The required details of the model equations can be found in

the original reference (Pyun et al., 1989).

The proposed ANNSA algorithm can be easily ex-

tended to handle this free ?nal time problem simply by

including the ?nal time tf as an additional decision vari-

able to be optimally determined. Thus, in this approach,

the number of decision variables is N + 1, where N is

the number of weights and biases of ANN chosen to

approximate the control function u(t). The ANN cho-

sen here is 1-3-1 which results in 11 decision variables.

Fig. 10 presents the optimal control pro?le obtained for

this problem. The feeding sequence is maximum-batch–

singular-batch and this is what can be expected from the

application of optimal control theory. The performance in-

dex obtained is 0.216218 and the ?nal time predicted is

12.65. The use of 1-5-1 ANN resulted in a performance

index of 0.216942 with a ?nal time of 12:65 h. These val-

ues compare favourably with the results reported by Pyun

et al. (1989). In fact, the performance index obtained in

this study is signi?cantly higher than the reported results

(PI = 0:20660;tf= 12:63).

6. Conclusion

This paper proposes a simple yet e?ective hybrid Ar-

ti?cial Neural Network Simulated Annealing (ANNSA)

algorithm for the solution of optimal control problems that

appear in chemical engineering applications. The ability of

a feedforward neural network with a single hidden layer to

approximate any arbitrary function has been exploited and

the optimal control problem is transformed into a nonlin-

ear programming problem where the decision variables are

the weights and biases of the networks. Simulation studies

Page 11

D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–31423141

0 0.050.1 0.15 0.2

0

2

4

6

8

10

12

14

16

18

20

Time (h)

u1

0 0.05 0.10.150.2

0

2

4

6

8

10

12

14

16

18

20

Time (h)

u2

0 0.05 0.10.150.2

0

2

4

6

8

10

12

14

16

18

20

Time (h)

u3

0 0.050.1 0.150.2

0

2

4

6

8

10

12

14

16

18

20

Time (h)

u4

Fig. 9. Optimal control pro?les for isothermal CSTR with four control variables (Case Study III): ANNSA with 1-5-4 ANN.

show that not many neurons in the hidden layer are neces-

sary for a reasonably good approximation of the control pro-

?les. The applicability of the proposed ANNSA method has

been demonstrated by solving four well-known challeng-

ing optimal control problems from the literature. The results

obtained for the case studies considered are in excellent

agreement with the published results. The computational

di?culties associated with the solution of singular optimal

control problems can be very easily avoided as the ANNSA

method can clearly identify the singular arcs in the opti-

mal control problems. Also, the e?ciency of the method is

not dependent on the choice of initial guesses of the deci-

sion variables. The present approach, as demonstrated in this

study, is very ?exible and can easily solve both multicontrol

and free ?nal time problems.

Notation

bi;out

bj

F

m

N

Neval

NH

NS;NT;NC SA parameters

ns

number of switches in the control policy

PI performance index

P

vector of weights and biases for ANN

r uniform random number in [ − 1;1]

t time, h

bias for ith output neuron

bias for jth hidden neuron

function represented by ANN

number of control variables/output neurons

number of weights and biases of ANN

number of function evaluations

number of hidden layer neurons in ANN

Page 12

3142 D. Sarkar, J. M. Modak/Chemical Engineering Science 58 (2003) 3131–3142

02468 101214

0

0.05

0.1

0.15

0.2

Time (h)

Feed Rate (L/h)

Fig. 10. Optimal feed rate pro?le for biphasic growth of yeast (Case

Study IV): ANNSA with 1-3-1 ANN.

T0

u

ˆ u

Wij

initial temperature of SA

control variable vector

scaled control variable vector

weight connecting ith output neuron to jth

hidden neuron

state variable vectorx

Greek letters

?

?

?

?

’

?

?j

temperature reduction factor

SA convergence parameter

step length vector

performance index

sigmoidal activation function

scaled time

weight connecting input neuron to jth hidden

neuron

Subscripts

f

max

min

?nal

maximum

minimum

References

Banga, J. R., Irrizarry-Rivera, R., & Seider, W. D. (1998). Stochastic

optimisation for optimal and model-predictive control. Computers &

Chemical Engineering, 22(4/5), 603–612.

Baughman, D. R., & Liu, Y. A. (1995). Neural networks in bioprocessing

and chemical engineering. San Diego, CA: Academic Press.

Cardoso, M. F., Salcedo, R. L., de Azevedo, S. F., & Barbosa, D.

(2000). Optimization of reactive distillation processes with simulated

annealing. Chemical Engineering Science, 55, 5059–5078.

Carrsco, E. F., & Banga, J. R. (1997). Dynamic optimisation of

batch reactors using adaptive stochastic algorithms. Industrial and

Engineering Chemistry Research, 36(6), 2252–2261.

Cybenko, G. (1989). Continuous valued neural networks with two hidden

layers are su?cient. Mathematics of Control, Signals, and Systems,

2, 303–314.

Dolan, W. B., Cummings, P. T., & Le Van, M. D. (1989). Process

optimisation via simulated annealing: Application to network design.

A.I.Ch.E. Journal, 35(5), 725–736.

Floquet, P., Pibouleau, L., & Domenech, S. (1994). Separation sequence

synthesis: How to use simulated annealing procedure?. Computers and

Chemical Engineering, 18(11/12), 1141–1148.

Gear, C. W. (1971). Numerical initial value problems in ordinary

di?erential equations. Englewood Cli?s, NJ: Prentice-Hall, Inc.

Go?e, W. L., Ferrier, G. D., & Rogers, J. (1994). Global optimisation of

statistical functions with simulated annealing. Journal of Econometrics,

60, 65–99.

Hanke, M., & Li, P. (2000). Simulated annealing for the optimisation

of batch distillation processes. Computers and Chemical Engineering,

24, 1–8.

Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward

networks are universal approximators. Neural Networks, 2, 359–366.

Jayaraman, V. K., Kulkarni, B. D., Gupta, K., Rajesh, J., & Kusumaker,

H. S. (2001). Dynamic optimisation of fed-batch bioreactors using the

ant algorithm. Biotechnology Progress, 17, 81–88.

Kirkpatrick, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimisation by

simulated annealing. Science, 220, 671–680.

Lee, J., & Ramirez, W. F. (1994). Optimal fed-batch control of induced

foreign protein produced by recombinant bacteria. A.I.Ch.E. Journal,

40, 899–907.

Luus,R. (1990). Application

high-dimensional non-linear optimal control problems. International

Journal of Control, 52(1), 239–250.

Luus, R. (2000). Iterative dynamic programming. London, UK: Chapman

& Hall/CRC.

Mekarapiruk, W., & Luus, R. (2000). Optimal control by iterative dynamic

programming with deterministic and random candidates for control.

Industrial and Engineering Chemistry Research, 39, 84–91.

Na, J. G., Chang, Y. K., Chung, B. H., & Lim, H. C. (2002). Adaptive

optimisation of fed-batch culture of yeast by using genetic algorithms.

Bioprocess and Biosystems Engineering, 24, 299–308.

Oberle, H. J., & Sothmann, B. (1999). Numerical computation of optimal

feed rates for a fed-batch fermentation model. Journal of Optimisation

Theory and Applications, 100(1), 1–13.

Park, S., & Ramirez, W. F. (1988). Optimal production of secreted protein

in fed-batch reactors. A.I.Ch.E. Journal, 34(9), 1550–1558.

Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., & Mischenks,

E. F. (1962). The mathematical theory of optimal processes. English

translation by K.N. Trirogo?. New York: Wiley.

Pyun, Y. R., Modak, J. M., Chang, Y. C., & Lim, H. C. (1989).

Optimisation of biphasic growth of saccharomyces carlsbergenesis in

fed-batch culture. Biotechnology and Bioengineering, 33, 1–10.

Rao, S. N., & Luus, R. (1972). Evaluation and improvement of control

vector iteration procedures for optimal control problems. The Canadian

Journal of Chemical Engineering, 50, 777–784.

Ray,W.H.(1981).Advancedprocesscontrol.NewYork:McGraw-HillInc.

Roubos, J. A., van Straten, G., & van Boxtel, A. J. B. (1999). An

evolutionary strategy for fed-batch bioreactor optimisation; concepts

and performance. Journal of Biotechnology, 67(2/3), 173–187.

Shukla, P. V., & Pushpavanam, S. (1998). Optimisation of biochemical

reactors—ananalysisof di?erent

operation. Chemical Engineering Science, 53(2), 341–352.

Simutis, R., & Lubbert, A., (1997). A comparative study on random search

algorithms for biotechnical process optimization. J. Biotechnology, 52,

245–256.

Stutts, B. E. (1983). An improved algorithm for singular control problems

with applications to the optimisation of the fed-batch penicillin fermen-

tation. Ph.D. thesis, Purdue University, West Lafayatte, Indiana, USA.

Tholuder, A., & Ramirez, W. F. (1997). Obtaining smoother singular arc

policies using a modi?ed iterative dynamic programming algorithm.

International Journal of Control, 68(5), 1115–1128.

of dynamic programmingto

approximationsoffed-batch