Content uploaded by Khaled Elleithy

Author content

All content in this area was uploaded by Khaled Elleithy

Content may be subject to copyright.

Content uploaded by Khaled Elleithy

Author content

All content in this area was uploaded by Khaled Elleithy

Content may be subject to copyright.

Content uploaded by Khaled Elleithy

Author content

All content in this area was uploaded by Khaled Elleithy

Content may be subject to copyright.

Keywords: Parallel and distributed systems, discrete event

simulation, null message algorithm.

Abstract

Null message algorithm is an important conservative time

management protocol in parallel discrete event simulation

systems for providing synchronization between the distributed

computers with the capability of both avoiding and resolving

the deadlock. However, the excessive generation of null

messages prevents the widespread use of this algorithm. The

excessive generation of null messages results due to an

improper use of some of the critical parameters such as

frequency of transmission and Lookahead values. However, if

we could minimize the generation of null messages, most of the

parallel discrete event simulation systems would be likely to

take advantage of this algorithm in order to gain increased

system throughput and minimum transmission delays. In this

paper, a new mathematical model for optimizing the

performance of parallel and distributed simulation systems is

proposed. The proposed mathematical model utilizes various

optimization techniques such as variance of null message

elimination to improve the performance of parallel and

distributed simulation systems. For the sake of simulation

results, we consider both uniform and non-uniform distribution

of Lookahead values across multiple output lines of an LP.

Our experimental verifications demonstrate that an optimal

NMA offers better scalability in parallel discrete event

simulation systems if it is used with the proper selection of

critical parameters.

1. INTRODUCTION

While there is a considerable literature exploring how poor

selection of critical parameters might results poor performance

of PDES systems [8, 12], surprisingly little work has examined

how critical parameters impact on the performance of PDES

systems. These research works indicate the strong relationship

among many critical parameters such as Lookahead and

frequency of transmission that one may use to quantify the

impact of these parameters on the PDES performance. None of

these research works, however, evaluate the determinants of

the critical parameters to the performance of PDES systems.

This paper presents a mathematical model to optimize the

performance of PDES systems by minimizing the null message

transmission across each LP using various optimization

techniques.

In parallel discrete event simulation (PDES) systems, the

distributed discrete events need to be tightly synchronized with

each other in order to work simultaneously on different parts of

a common task. However, if these discrete events are not

properly synchronized, the performance of a PDES

environment may degrade significantly [2]. Time management

algorithms (TMA) are, therefore, required to ensure that the

execution of a PDES is properly synchronized. In general,

synchronization protocols can be categorized into two different

families: conservative and optimistic. In optimistic algorithm,

both deadlock detection and recovery occur at run time.

However, if it is used in a wide range parallel network, each

logical process (LP) may experience longer transmission

delays at run time [13]. On the other hand, conservative

protocols fundamentally maintain causality in event execution

by strictly disallowing the processing of events out of

timestamp order. In order to avoid and resolve the deadlock

situations, each LP needs to exchange time stamp information

with the other neighboring LPs [1, 3]. Examples of

conservative mechanisms include Chandy, Misra and Byrant's

NMP [6], and Peacock, Manning, and Wong [11] avoided

deadlock through null messages.

Conservative TMA can be further classified as synchronous

and asynchronous protocols [1]. Synchronous algorithm uses

global synchronization mechanism to determine the minimum

time stamp of each incoming event for an LP. On the other

hand, NMA is an example of an asynchronous conservative

algorithm that does not require global synchronizations. The

primary problem associated with null messages is that if their

timestamps are chosen inappropriately, the simulation becomes

choked with null messages and performance suffers. Some

intelligent approaches to null message generation include

generation on demand [8], and generation after a time-out [5].

Some earlier research on discrete event simulation has focused

on variants of null message protocol (NMP, with the objective

of reducing the high null message overhead. For instance, Bain

A New Mathematical Model for Optimizing the Performance of Parallel and

Discrete Event Simulation Systems

Syed S. Rizvi and Khaled. M. Elleithy

Computer Science and Engineering Department

University of Bridgeport

Bridgeport, CT 06601

{srizvi, elleithy}@bridgeport.edu

Aasia Riasat

Department of Computer Science

Institute of Business Management

Karachi, Pakistan 78100

aasia.riasat@cbm.edu.pk

and Scott [4] attempt to simplify the communication topology

to resolve the problem of transmitting redundant null messages

due to low Lookahead cycles. Other recent developments [10]

have focused on incorporating knowledge about the LP into the

synchronization algorithms. Cota and Sargent [7] focused on

the skew in simulation time between different LPs by

exploiting knowledge about the LPs and the topology of the

interconnections.

Although, much research has been done to evaluate the

performance of conservative NMA for inefficiencies and

transmission overhead [3, 8, 12], none of them suggest any

potential optimization for the NMA. Reference [12] proposed a

new approach that shows relationships between many

parameters to quantify the performance of PDES system

running under NMA. It has been shown that the selection of

values for several critical parameters such as the values for

Lookahead, null message ratio (NMR), and frequency of

transmission plays an important role in the generation of null

messages [12]. If these values are not properly chosen by a

simulation designer, the result will be an excessive number of

null messages across each LP. This situation gets more severe

when the NMA needs to run to perform a detailed logistics

simulation in a distributed environment to simulate a huge

amount of data [9]. This paper presents a mathematical model

that implements many optimization techniques to optimize the

performance of NMA by minimizing the exchange of null

messages across the LPs. A significant improvement is

measured in the performance of PDES system in terms of

reduced execution speed and transmission delays.

The rest of the paper is organized as follows. Section 2

presents the proposed mathematical model. Section 3 provides

the implementation of various optimization techniques on the

NMA for improving the performance of PDES systems. The

numerical and simulation results are presented in Section 4.

Finally, we conclude in Section 5.

2. PERFORMANCE OPTIMIZATION THROUGH

PROPOSED MATHEMATICAL MODEL

Some of the important model variables, along with their

definition, are listed in Table I. For the sake of mathematical

model, we assume that the value of Lookahead may change

during the execution of a Lookahead period. However, a

sudden increase or decrease in the values of Lookahead during

the simulation can not be accepted. In addition, we assume that

each LP is initialized with a constant event arrival. However,

as the simulation progresses, we use both uniform and non-

uniform distribution of Lookahead values across multiple

output lines of each LP. For the frequency of message

transmission, we assume that all messages are equally

distributed among the LPs. Unless otherwise stated, we use the

term all messages to refer to both null and event messages.

Finally, we assume that a fixed size message is transmitted

between LPs.

Our proposed mathematical model is based on the internal

architecture of an LP as shown in Fig. 1 and Fig. 2. The

advancement in simulation time can be defined as a ratio of

performance to the event arrival rate. The number of event

messages processed per second per LP is represented by P,

where as the occurrence of the number of events per simulation

second is refereed as an even arrival rate and it is represented

by ρ. This leads us to the following mathematical expression of

the relative speed for advancement:

( )

Msg

T

s

PE

(1)

Taking this into account, we can give the following

hypothesis for approximating the number of null messages

transmitted per LP: “If we assume that we have an average

value of Lookahead (L) which associates with one of the

output lines of an LP, then P can be approximated as”:

1

Msg

P E

L

≅

(2)

Combining (1) and (2) yields the estimated number of null

messages transmitted per LP that has only one output line as

shown in (3).

( )

1 1

S

Msg S

LP

Msg

T

Null E T

L E L

(3)

Furthermore, if we assume that we have O number of output

lines attached with each LP with the uniform distribution of

Lookahead value on each output line, then (3) can be further

generalized for O number of output lines per LP as follows:

( )

S

Msg S

LP

Msg

T

O O

Null E T

L E L

(4)

where O represents the total output lines per LP.

It should be noted that (4) represents total number of null

messages transmitted per LP via O number of output lines to

the neighboring LPs when the distribution of L is assumed to

be uniform per output lines. If we assume that we have m

number of total LPs present in a system where each LP has O

number of output lines, then this allows us to extend (4) and

generalize it for m number of LPs present in distributed

simulation as shown in (5). It can be evident that (5) gives total

TABLE I

System Parameter Definition

Parameter Definition

P

Computation required for processing an event per second

ρ

Event arrival rate (events per second)

MRT

Minimum receiving time

MST

Minimum sending time

L

Lookahead

STA

Simulation time advancement

F

T

Frequency of transmission

T

Null

Timestamp of a null message

T

S

Current simulation of a LP

T

Total

Total simulation time in seconds

number of null messages exchange among all LPs present in

the system.

( )

S

Msg S

m LP

Msg

T

O O

Null E m T m

L E L

−

(5)

where the term

O L

in (5) shows a uniform distribution of

Lookahead value for O number of output lines per LP and the

term m represents total number of LPs in the system.

The assumption of uniform distribution of Lookahead

among O output lines of an LP simplifies the procedure for

computing the number of null messages transmitted per LP to

other neighboring LPs. However, the values for Lookahead

may change during the execution of a Lookahead period that

makes the uniform distribution assumption of Lookahead a

little unrealistic. Based on this argument, we can rewrite (4) as:

( )

1 1

1

O O

Msg S

S

LP

i i

i Msg i

ET

Null T

L E L

= =

∑ ∑

(6)

It should be noted that (6) represents the total number of null

messages transmitted per LP to other neighboring LPs via O

number of output lines where each line can have a different

Lookahead value.

If we assume that the model is partitioned into m number of

total LPs where each LP can have at most O number of output

lines, then this allows us to extend (6) and generalize it for m

number of LPs. This generalization can be expressed in (7).

( )

1 1 1 1

1

m O m O

Msg S

S

m LP

h i h i

hi Msg hi

ET

Null T

L E L

−

= = = =

∑∑ ∑∑

(7)

It can be evident that (7) gives total number of null messages

exchange among all LPs present in the system.

3. PROPOSED OPTIMIZATION TECHNIQUES FOR

NMA

In this section, we first derive a closed form mathematical

expression for both frequency of transmission and the variance

of null message elimination that can be further used to

determine the reduction in the null message traffic in the

presence of deadlock. The derived closed form expression uses

the simple concept of frequency of transmission described in

[12] to minimize the exchange of null messages across the LPs.

In addition, we implement the optimization technique via

variance of null message elimination.

3.1. Optimization Via Frequency of Transmission

Instead of sending null messages after processing each event

on each output line of an LP, it should be transmitted with

respect to a certain frequency of transmission. This frequency

of transmission (FT) is a fixed amount of time and it should be

measured in simulation second per second. In other words, the

Lookahead value which is associated with one or more output

lines can be approximated as the frequency of transmission per

output line of an LP. The above argument yields the following

approximation for FT in term of the Lookahead value.

1

2 2

Msg

Msg T

T

E L

E L F

F L

≅⇒≅

(8)

Substituting the value of (8) into (3), we get,

( )

1 1

2 2

S

S

LP

T T

Msg

Msg

T

Null E T

E

F F

(9)

Equation (9) can be generalized for O number of output

lines per LP when the numbers of null messages are generated

with a certain frequency of transmission. In other words, the

expected rate (i.e., FT) at which null messages may generate

per output line per LP can be roughly estimated as a percentage

of the Lookahead values. This expected rate per output line per

LP results (10) as follows:

( )

%

2 2

S

Msg S T

LP

Msg

T T

TO O

Null E T where F L

E

F F

→

(10)

Equation (10) gives an estimated number of null messages

transmitted by single LP that has O number of output lines

where each line carry an equal percentage of the Lookahead

value in terms of a fixed frequency of transmission per output

Fig.1. Internal architecture of an LP

Fig.2. m number of logical processes with I number of input

queues and O number of output queues per LP.

line. In addition, if we assume that the system consists of m

number of total LPs where each LP has fixed number of output

lines, then (10) can be further extended for m number of LPs.

This generalization results (11) as follows:

( )

( )

( ) ( )

%

2 2

S

Msg T S T

m LP

Msg

T

T

Null E m O F T m O F

E

where F L

−

× ×

→

(11)

where the denominator of (11) (i.e.,

T

F

O) represents a

uniform rate of null message transmission per output line.

Based on (6), we can conclude that a non uniformity in null

message algorithm results non linear generation of null

messages. In other words, the approximation of null messages

can be well optimized when a non uniform transmission rate is

considered. Based on this argument, a mathematical expression

can be derived for O number of output lines where each line

may carry a different frequency of transmission.

( )

1 1

%

1

2 2

O O

Msg S

S

LP

i i

Msg

Ti Ti

Ti i

ET

Null T

E

F F

where F L

= =

→

∑ ∑

(12)

Furthermore, (12) can be further extended and generalized

for m number of LPs where each LP can have at most O

number of output lines. This generalization can be expressed in

(13).

( ) ( )

( )

( )

( )

( ) ( )

1 1 1 1

%

1

2

2

m O m O

Msg S Msg S

m LP T ki

k i k i T ki

T ki ki

Null E F T E T F

where F L

−

= = = =

→

∑∑ ∑∑

(13)

3.2. Optimization Via Variance for Null Message

Elimination in NMA

Also, in this scenario, it is essential to cancel out the

unnecessary generation of null messages. To consider and

analyze the effect of null message elimination on the

performance of PDES systems, we introduce variance as a

variable quantity. Variance represents the probability of

cancellation of unnecessary null messages. The value of

variance may exist between 0 and 1 (i.e., it can not be one,

since 1 represents that all generated null messages cancelled

with the maximum probability). It should also be subtracted

from 1, so that we can show that increase in variance causes a

decrease in the over all null messages where as a decrease in

variance causes an increase in null messages. If we consider

variance as 0, then it should give us the same results that we

could achieve with out using variance. In order to reflect the

variance of null message cancellation, we can rewrite (11) for

m number of LPs with the uniform distribution of null message

transmission per output line as follows:

( )

( )

( )

%

2 (1 )

(1 ) 0 1

2

S

Msg T

m LP

Msg

S T

T

T

Null E m O F E

O

T m where F L and

F

σ

σ σ

−

× −

× − → ≤ <

(14)

where

σ

represents probability of null message cancellation.

The same concept of null message cancellation can be

implemented with a simulation model where the Lookahead

values are non-uniformly distributed among O number of

output lines. This leads us to the following modification in

(14):

( )

( ) ( )

( ) ( )

1 1 1 1

%

1

(1 ) (1 )

2 2

0 1

m O m O

Msg S

S

m LP

k i k i

Msg

T T

ki ki

Tki ki

ET

Null T

E

F F

where F L and

σ σ

σ

−

= = = =

− −

→ ≤ <

∑∑ ∑∑

(15)

4. PERFORMANCE ANALYSIS OF THE PROPOSED

MATHEMATICAL MODEL

For the sake of performance analysis, we simulate 6 different

cases. The system is modeled in C++

4.1. CASE-I: Multiple Output Lines per LP

Using (4) [Null (LP) = Ts (O/L)], Fig.3 shows the null

message transmission with the following simulation

parameters: simulation time = 500 sec, L is uniformly

distributed per output line (O). The number of output line may

vary from 0 to 8 for all cases as show in Fig.3. Both numerical

and simulation results present a comparison of null message

transmission per LP versus multiple output lines.

4.2. CASE-II: Multiple LPs with Multiple Output Lines

per LP

In CASE-II, we assume that we have multiple LPs with O

output lines (fixed per LP). Let the output lines per LP is 4

with the simulation Time (Ts) of 500 sec. Using (5)

( )

S

Msg S

m LP

Msg

T

O O

Null E m T m

L E L

−

, Fig.4 shows the null

234567 8

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Output lines (O) Per LP

Null Message Transmission Per LP

Null(LP) with L=0.2

Null(LP) with L=0.4

Null(LP) with L=0.6

Null(LP) with L=0.7

Fig3. Multiple output lines per LP versus null message

transmission per LP

message transmission with the following simulation

parameters: simulation time = 500 sec, L is uniformly

distributed per output lines (O), the output lines are assumed to

be fixed for each LP (O = 4). The numbers of LPs are varied

from 1 to 10 as show in Fig.4.

4.3. CASE III: Multiple Output Lines per LP with Non-

Uniform Distribution of Lookahead

For this simulation, we assume that we have single LP that

has O number of output lines where each output line of an LP

can have different value of Lookahead (L). Using (6), Fig.5

shows the null message transmission with the following

simulation parameters: simulation time = 500 sec, L is non-

uniformly distributed per output lines (O). The numbers of

output lines are varied from 1 to 10 as show in Fig.5. Also, it

should be noted that the value of Lookahead is chosen

randomly within the range of 0 to 1 and assigned to each

output line at run time.

4.4. CASE-IV: Multiple LPs with Multiple Fixed Output

Lines where each Output line can have Different

Lookahead Value

For this simulation, we assume that we have multiple LPs

that can have fixed number of output lines where each line of

an LP can have different value of Lookahead (L). Using (7),

Fig.6 shows the null message transmission with the following

simulation parameters: simulation time = 500 sec, L is non-

uniformly distributed per output lines (O). The numbers of LPs

are varied from 1 to 20 as show in Fig.6. Also, it should be

noted that the value of m and O are both varying quantity for

this particular scenario.

5. CONCLUSION

We have proposed a mathematical model to predict the

optimum values of critical parameters that have great impact

on the performance of NMA. The proposed mathematical

model provides a quick and practical way for simulation

designers to predict whether a simulation model has potential

to perform well under NMA in a given simulation environment

by giving the approximate optimal values of the critical

parameters. We have experimentally verified that if critical

parameters, specifically the Lookahead value, are chosen

intelligently, we can limit the transmission of null messages

among the LPs.

REFERENCES

[1] R. M. Fujimoto, “Distributed Simulation system,” preceding of

the 2003 winter simulation conference. College of Computing,

Georgia Institute of Technology, Atlanta.

[2] Y.M. Teo, Y.K. Ng and B.S.S. Onggo, “Conservative

Simulation using Distributed Shared Memory,” Proceedings of

the 16th Workshop on Parallel and Distributed Simulation

(PADS-02), IEEE Computer Society, 2002.

2 4 6 8 10 12 14 16 18 20

400

600

800

1000

1200

1400

1600

1800

2000

Number Of output lines (O) per LP

Null Message Transmission per LP

Null Messages where 0<L<1 (100 Runs)

Null Messages where 0<L<1 (300 Runs)

Null Messages where 0<L<1 (500 Runs)

Null Messages where 0<L<1 (700 Runs)

Fig5. Multiple Output Lines per LP with Non-

Uniform

Distribution of Lookahead versus null message transmission

2 4 6 8 10 12 14 16 18 20

600

700

800

900

1000

1100

1200

1300

1400

1500

Number Of LPs

Null Message Transmission

4 output lines per LP where 0<L<1 (100 Runs)

6 output lines per LP where 0<L<1 (300 Runs)

8 output lines per LP where 0<L<1 (500 Runs)

10 output lines per LP where 0<L<1 (700 Runs)

Fig.6. Multiple LPs and Multiple Fixed Output Lines

with

non-

uniform distribution of Lookahead value versus null

message transmission

2 3 4 5 6 7 8 9 10

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Number Of LPs

Null Message Transmission

Null(m-LP) with L=0.2

Null(m-LP) with L=0.3

Null(m-LP) with L=0.4

Null(m-LP) with L=0.6

Null(m-LP) with L=0.7

Fig4

. Multiple LPs with Fixed Output Lines per LP versus

null message transmission

[3] B. R. Preiss, W. M. Loucks, J. D. MacIntyre, J. A. Field, “Null

Message Cancellation in Conservative Distributed Simulation,”

Distributed Simulation 91 Proceedings of the SCS

Multiconference on Advances in Parallel and Distributed

Simulation, 1991.

[4] W. L. Bain, and D. S. Scott, "An Algorithm for Time

Synchronization in Distributed Discrete Event Simulation",

Proceedings of the SCS Multiconference on Distributed

Simulation, 19, 3 (February), pp. 30-33, 1988.

[5] N. J. Davis, D. L. Mannix, W. H. Shaw, and Hartrum, T. C.,

‘‘Distributed Discrete-Event Simulation using Null Message

Algorithms on Hypercube Architectures,’’ Journal of Parallel

and Distributed Computing, Vol. 8, No. 4, pp. 349-357, April

1990.

[6] K. M. Chandy and J. Misra, "Distributed Simulation: A case

study in design and verification of distributed programs", IEEE

Transactions on Software Engineering, SE-5:5, pp. 440-452,

1979.

[7] B. A. Cota and R. G. Sargent, ‘‘An Algorithm for Parallel

Discrete Event Simulation using Common Memory,’’ Proc.

22nd Ann. Simulation Symp., pp. 23-31, March 1989.

[8] J. K. Peacock, J. W. Wong, and E. Manning, ‘‘Synchronization

of Distributed Simulation using Broadcast Algorithms,’’

Computer Networks, Vol. 4, pp. 3-10, 1980.

[9] L. A. Belfore, S. Mazumdar, and S. S. Rizvi et al., “Integrating

the joint operation feasibility tool with JFAST,” Proceedings of

the Fall 2006 Simulation Interoperability Workshop, Orlando Fl,

September 10-15 2006.

[10] D. M. Nicol and P. F. Reynolds, ‘‘Problem Oriented Protocol

Design,’’ Proc. 1984 Winter Simulation Conf., pp. 471-474,

Nov. 1984.

[11] J. K. Peacock, J. W. Wong, and E. Manning, ‘‘A Distributed

Approach to Queuing Network Simulation,’’ Proc. 1979 Winter

Simulation Conf., pp. 39 9-406, Dec. 1979.

[12] Syed S. Rizvi, K. M. Elleithy, Aasia Riasat, “Minimizing the

Null Message Exchange in Conservative Distributed

Simulation,” International Joint Conferences on Computer,

Information, and Systems Sciences, and Engineering, CISSE

2006, Bridgeport CT, pp. 443-448 ,December 4-14 2006,

[13] Syed S. Rizvi, Khaled M. Elleithy, and Aasia Riasat, “Trees and

Butterflies Barriers in Mattern’s GVT: A Better Approach to

Improve the Latency and the Processor Idle Time for Wide

Range Parallel and Distributed Systems”, IEEE International

Conference on Information and Emerging Technologies (ICIET-

2007), July 06-07, 2007, Karachi, Pakistan.