Content uploaded by Yousry Y. Azmy

Author content

All content in this area was uploaded by Yousry Y. Azmy on Jun 30, 2015

Content may be subject to copyright.

The Three-Dimensional, Discrete Ordinates Neutral Particle

Transport Code TORT: An Overview*

Y. Y. Azmy

Oak Ridge National Laboratory**

Oak Ridge, Tennessee 37831-6363

"The submitted manuscript has been authored by a

contractor of the U. S. Government under contract

No.

DE-AC05-96OR22464. Accordingly, the U.S.

Government retains a nonexclusive, royalty- tree

license to publish or reproduce the publised form of

this contribution, or allow others to do so, for U.S.

Government purposes."

BBC

MASTER

to be presented at the

OECD/NEA meeting on 3D Deterministic Radiation Transport Computer Programs,

Feature, Applications and Perspectives

Paris,

France

December 2-3, 1996

* Research sponsored by the U.S. Department of Energy.

*Managed by Lockheed Martin Energy Research Corp. for the U. S. Department of Energy under

Contract DE-AC05-96OR22464.

THE THREE-DIMENSIONAL, DISCRETE ORDINATES NEUTRAL PARTICLE

TRANSPORT CODE TORT: AN OVERVIEW*

Y. Y. Azmy

Oak Ridge National Laboratory

P.O.

Box 2008, MS 6363

Oak Ridge, TN 37831

Abstract

The centerpiece of the Discrete Ordinates Oak Ridge System (DOORS), the three-dimensional

neutral particle transport code TORT is reviewed. Its most prominent features pertaining to large

applications, such as adjustable problem parameters, memory management, and coarse mesh

methods, are described. Advanced, state-of-the-art capabilities including acceleration and multipro-

cessing are summarized here and detailed in other papers in these Proceedings. Future enhancement

of existing graphics and visualization tools is briefly presented.

Introduction

The Discrete Ordinates Oak Ridge System (DOORS) is comprised of several computer codes

developed over the years at Oak Ridge National Laboratory to solve a wide variety of neutral parti-

cle transport problems arising in applications. The first release of DOORS 3.1 in 1995 was charac-

terized by a modernized installation procedure and an expanded list of peripheral codes included in

the distribution, as well as other new features.

1

"

2

Bringing the member codes together in DOORS is

the first step in establishing smoother connections between them which will eventually be manipu-

lated via a user friendly Graphical User Interface (GUI).

As the power and capacity of electronic computers expanded over the past three decades the

work horse of the transport calculations evolved in dimensionality from one-, to two-, to three-

dimensional. Time consuming cross section generation is often still performed using one-

dimensional models, and two-dimensional models still suffice for many applications. Hence

DOORS includes codes for these purposes; the focus of this paper, however, is the three-

dimensional code TORT.

3

TORT was conceived in the mid eighties to compute radiation dose profiles in large multistory

concrete buildings. While largely based on its two-dimensional predecessor, DORT, it quickly

evolved in an individual fashion suited to the difficulties associated with the problems it is applied

to,

primarily size. Thus high order, coarse mesh methods, vectorization of the mesh sweep, discon-

tinuous mesh, a^nong others, are features developed and implemented exclusively for TORT.

3

These

Managed by Lockheed Martin Energy Research, Inc. under contract DE-AC05-96OR22464 with the U.S. Department of

Energy.

DISCLAIMER

This report was prepared as an account of work sponsored by an agency of the

United States Government Neither the United States Government nor any agency

thereof, nor any of their employees, makes any warranty, express or implied, or

assumes any legal liability or responsibility for the accuracy, completeness, or use-

fulness of any information, apparatus, product, or process disclosed, or represents

that its use would not infringe privately owned rights. Reference herein to any spe-

cific commercial product, process, or service by trade name, trademark, manufac-

turer, or otherwise does not necessarily constitute or imply its endorsement, recom-

mendation, or favoring by the United States Government or any agency thereof.

The views and opinions of authors expressed herein do not necessarily state or

reflect those of the United States Government or any agency thereof.

DISCLAIMER

Portions of this document may be illegible

in electronic image products. Images are

produced from the best available original

document

and other unique features of the code made it an invaluable computational tool for large problems

in a wide variety of applications. A high level of confidence in TORT's reliability and computa-

tional efficiency has been established over the years through the experiences of many researchers

and engineers working in different areas.

While TORT is a general purpose transport code it has gained particular notoriety for accu-

rately solving large problems with complex configurations. At present the largest TORT application

employs 3.6 million computational cells, Sj

6

angular quadrature, P

3

anisotropic scattering expansion,

and 11 energy groups, a monumental challenge to the computational resources typically available at

research institutions. The code's production attributes, such as robustness, efficiency, accuracy, and

reliability, resulted in its proliferation into many areas of application new to deterministic transport

methodologies. These include medical applications,

4

*

5

charged particle transport,

6

and time depen-

dent calculations,

7

in addition to the traditional reactor physics and shielding applications; see for

example Refs. 8,9.

In the remainder of this paper we present an overview of TORT emphasizing its most salient

features that make it particularly suitable for large production level applications. We start with a

brief description of DOORS and its member code, then we discuss adjustable problem parameters

including discontinuous mesh, and the memory management capability. The three spatial approxi-

mations available in TORT are reviewed next, then acceleration schemes for the iterative process

are described, followed by a brief section on multiprocessing with TORT. The last section is dedi-

cated to graphical routines currently available and visualization tools planned for the future.

Discrete Ordinates Oak Ridge System (DOORS)

DOORS is a collection of codes built around the discrete-ordinates codes ANISN, DORT, and

TORT gives the user a wide range of capabilities to pre- and post-process data in transition from

one computation to the next. These codes represent a total investment of many man-years of effort

contributed by many researchers over the past three decades. While most codes included in the

present version of DOORS are in their original condition, they have been packaged and will be

maintained in unisome in the future. DOORS also introduced modern maintenance and installation

procedures, e.g. makefiles, that facilitate updating activities and extend the package's compatibility

into the future. ORNL supports DOORS on UNIX-based platforms, Cray supercomputers, and a

variety of workstations; PC versions of some of the member codes have been reported and are

available from RSICC.

A complete list of the codes comprising the DOORS package at present is shown in Table 1.

Among these perhaps the best known are the one-, and two-dimensional sister codes ANISN, and

DORT, respectively. While TORT has a one-, and two-dimensional solution capability, these codes

are easier to use, and have additional features, e.g. some curvilinear geometry options, which have

been more thoroughly tested over the years. Furthermore, because their lower dimensionality is

hardwired into them, ANISN and DORT utilize computational resources more efficiently and are

more likely to provide faster execution. For this reason they are often used in scoping calculations

that require repetitive solution of slightly differing problem configurations. In this regard the

adjoint capability of all DOORS transport codes sometimes aides the search for optimal

configurations.

5

The TORSED and TORSET codes implement splicing and bootstrapping techniques that per-

mit the solution of problems that are too large to solve with TORT in their entirety. Essentially

these methods amount to a split of the problem domain into two, or more, subdomains that are

loosely coupled in only one direction from the source, primary, to the observable, secondary,

configurations, with weak feedback. For details of these techniques and their implementation in

important example which has proven extremely valuable in solving very large problems with sub-

stantial geometric detail is the discontinuous mesh option in TORT. This feature allows the user to

employ more computational cells where fine geometric structure exists, and fewer cells elsewhere.

This is achieved by allowing the user, within some constraints, to adjust the number and boundaries

of individual cells within a row, not necessarily coinciding with those in adjacent rows. The fact

that the solution algorithm in TORT reduces the most general problem to a sequence of sweeps

along a row of computational cells in the x -dimension in a given direction allows each row to be

processed individually then the boundary angular flux incoming to adjacent rows are inter- or extra-

polated to that row's cell structure.

Other adjustments to problem parameters not related to phase space size include, for example,

setting iteration number by group, etc. These typically pertain more to reducing execution time by

avoiding tight convergence of groups that are inconsequential to the purpose of the computation.

Recent research aimed at providing TORT with the capability to account for heterogeneous

material, combinatorial geometry objects by manipulating the spatial weights in the discrete-van able

equations is detailed in Ref. 11.

Memory Management

In spite of the flexibility afforded the user in concentrating the computational effort at regions

of high sensitivity to the level of detail it is customary in scientific research and engineering design

to constantly push the capability limits in order to make progress. Thus the insatiable demand for

finer detail kept pace with the phenomenal growth in memory size and external storage options

available. Virtual memory is a generic solution to this problem available on many platforms that is

designed to conduct the I/O activity at the operating system level thereby relieving the programmer

from the burden of foreseeing and accommodating a wide variety of hardware configurations and

run time conditions. Nevertheless, the price of this flexibility is paid by the user in the form of

long execution times because the generic I/O scheme does not take into account natural breakpoints

in the solution algorithm of transport problems.

In recognition of this fact, and also to accommodate a variety of platforms with a disparate

range of memory sizes TORT attempts to meet a memory objective set by the user at run time.

First TORT tries to fit the entire problem in the memory objective; if this is not possible it attempts

to it one group at a time within the memory objective and I/O flux and cross section data to scratch

files. If this too fails, the code breaks up the geometric configuration into blocks of planes, each of

which can fit within the memory objective, and I/O to scratch files is used to maintain and update

the data during the calculation. If a single plane does not fit in the memory objective TORT tries to

obtain additional memory beyond that specified by the memory objective, assuming it is less than

all that is available on the machine, and if this fails it informs the user of its attempts then ter-

minates execution unsuccessfully. Clearly this sequence of attempts is designed to minimize the

adverse effect of I/O on performance while enabling the solution of ever larger problems.

Spatial Discretization Methods

TORT is^ based on the discrete ordinates approximation of the independent angular variable,

and the multigroup discretization of the energy variable. Three approximation methods for the spa-

tial dependence are available in the code to accommodate a broad spectrum of applications.

The oldest is the 9-weighted method, which was originally implemented in DORT, represents

a whole set of methods parametrized by the single parameter 0 set by the user at run time in the

Table 1. Member Codes in the DOORS Package

Code

anisn

gbanisn

dort

tort

torset

torsed

visa

ale

g>P

grtuncl

falstf

bndrys

rtflum

isoplot

xtorid

jdos

drv

emp

rscors

Function

Solve one-dimensional transport problems

anisn with group band option for thermal upscatter

Solve two-dimensional transport problems

Solve three-dimensional transport problems

Couple primary to secondary tort calculations

Couple rz-dort to xyz-tort calculations

Prepare torsed input from dort output file

Maintenance of cross section library

Prepare cross section library

Estimate uncollided flux and first collided source

Project last collision source to point detector

Convert internal boundary flux to internal boundary source

Convert flux moment file across formats

Generate contour plot of flux or response

Extract planar slice from tort output to plot with isoplot

Execute sequence of calculations

Driver module called by jdos

Maintenance of code system*

Graphics library*

TORT see Ref. 10.

* Public domain software from Sandia National Laboratory

Adjustable Problem Parameters

Perhaps the factor that most contributes to the difficulty of transport calculations is the large

size of the discrete variable system of equations that must be solved numerically. This is a direct

consequence of the high dimensionality of phase space, a fact that is easily illustrated by consider-

ing a a steady state transport problem in <i-dimensional geometry. In such case the phase space is

of dimension Id: d variables representing physical space, d-\ representing the particles direction

of motion (discrete ordinates), and one energy variable. It is critical, therefore, to conserve discrete

variables as much as possible without jeopardizing the accuracy and reliability of the solution. This

is accomplished in TORT by permitting the user to adjust the level of detail in a variety of problem

parameters according to the anticipated local rate of change in the solution.

For exanfple, sharp flux and cross section anisotropies are more notable at the high energy end

of the spectrum. Hence TORT permits the user to select a high order angular quadrature set and

high order P, expansion of the cross sections in the high energy groups, and lower orders in the

low energy groups thereby reducing the problem size without sacrificing accuracy. Another

range [0,1). This is a weighted diamond difference method spanning the range from the diamond

difference scheme (optically thin cells) to the Step method (optically thick cells) where the weights

are computed to ensure positivity of the outgoing angular flux given positive incoming flux and

volumetric source. Due to the relative simplicity of its equations and its low order approximation,

the 9-weighted method is the least computationally intensive option among TORT's spatial approx-

imation methods. Also, it is the least accurate on a given mesh.

The Linear Nodal (LN) method was installed in TORT to enable using optically thick cells

while retaining high accuracy of the computed angular flux. This is achieved by computing the first

spatial moment of the angular and scalar flux, in addition to the average quantities computed in the

0-weighted method.

12

'

13

Many modifications to the original method have been implemented over

the years in order to improve method accuracy and efficiency, and solution positivity to the extent

that, at present, LN is the recommended method for most large applications with optically thick

regions in TORT.

The Linear Characteristic (LC) method

15

also computes the first spatial moment of the flux on

each cell's surfaces and volume using the exact characteristic paths from incoming to outgoing sur-

faces,

then projecting the resulting expression onto the basis functions (constant and linear). As far

as accuracy is concerned LC is competitive with LN with each method gaining an edge ov£r the

other for some, but not all, problems. LC executes faster than LN on scalar machines, but due to

the high level of vectorization of the LN it is about four times faster on Cray computers.

14

On a given mesh LN and LC run longer, require more memory, and consume larger disk

space but provide more accurate solutions than the 6-weighted method. However, for a fixed accu-

racy the latter method typically requires eight times as many computational cells than either of the

linear schemes. Ultimately for the same solution accuracy requirement, the linear methods end up

utilizing less computational resources, i.e. CPU time, memory, and disk space, than the 9-weighted

method.

14

Iteration Acceleration Methods

The recommended method for accelerating the iterative convergence of the inner iterations in

TORT is the Partial Current Rebalance (PCR) method.

3

This method is based on reinforcing the

balance of neutrons over each computational cell using the cell-surface partial currents resulting

from the latest mesh sweep. The discrete variable equation resulting from PCR has the same cell-

coupling stencil as a discretized cell-centered diffusion equation but does not necessarily possess

some of its important features like diagonal dominance, etc. The PCR matrix equation is solved via

a Successive Over-Relaxation (SOR) scheme with the relaxation factor computed numerically from

the SOR iterates.

For most TORT applications PCR has proven robust and efficient. However, recent advances

in the analysis of the spectral properties of iterative procedures for solving the transport equation

have provided the basis for powerful acceleration operators that, at least theoretically, exceed the

performance of PCR. The most notable example of such new methods is Diffusion Synthetic

Acceleration (DSA) whose spectral radius is bounded from above by 0.25 for model problem

configurations, i.e. homogeneous material composition and uniform mesh. The most serious limita-

tion of the class of unconditionally stable DSA operators as far as large three dimensional applica-

tions is concerned is that they are edge-centered. Since there are many more surfaces than compu-

tational cells the DSA matrix equation can be prohibitively large; this difficulty is compounded

further in high order methods, i.e. LN and LC, if the first spatial moments of the flux are to be

accelerated also.

A more general framework for acceleration schemes, the Adjacent-cell Preconditioning

method, has been implemented recently in TORT and is reviewed in Ref. 15. AP is cell-centered

and has the same coupling scheme as a discretized cell-centered diffusion equation but its elements

are not based on the diffusive properties of a computational cell. Rather, the preconditioner ele-

ments are set to provide a vanishing spectral radius of the flat eigenmode of the homogeneous

model problem, with reciprocal averaging across material heterogeneity interfaces. Therefore, the

preconditioning stage of the iterative process is comprised of a system of discrete-variable equations

that is amenable to solution via the same SOR routines in TORT used to solve the PCR equations.

Testing of the AP in TORT demonstrated its effectiveness in reducing the number of iterations

required to achieve convergence for all members of the Burre Suite of Test Problems (BSTeP) cov-

ering a wide range in parameter space. In fact AP converged in fewer iterations than PCR or

TWODANT's DSA for the vast majority of cases comprising BSTeP. However, these tests illus-

trated the deterioration in spectral properties in cases with sharp material discontinuity contradicting

behavior predicted by the homogeneous model problem analysis. Furthermore, this undesirable

behavior seems to be commonly shared with other acceleration schemes including PCR and DSA.

Analysis of a model problem with material discontinuity, the Periodic Horizontal Interface (PHI),

has been analyzed to verify and investigate this phenomenon.

15

Multiprocessing

A multitasking capability at the macrotasking level is available in TORT for execution on

multiprocessor Cray computers running the UNICOS operating system. It is based on a coarse-

grained angular domain decomposition which typically produces good parallel efficiency due to the

relatively large computation load to parallelization overhead ratio. Since angular domain decompo-

sition in Cartesian geometry is intrinsic there is a one-to-one correspondence (within arithmetic pre-

cision) between the sequential and parallel intermediate and final results, so that the number of

iterations required to achieve convergence is independent of the number of concurrent processes.

The multitasking option, selected by the user at run time, has been tested on Cray Y/MP, C90, and

J90 systems and have exhibited significant wall clock speedup factors even for modestly large prob-

lems.

The problem with quantifying parallel performance on time-shared computers such as the

Cray is its sensitivity to machine loading during execution. Nevertheless it was possible to quantify

the parallelization overhead for two test problems and conclude the necessity to reduce it sharply.

Recently, a performance model was constructed and validated ibr the parallelization overhead

as a function of the number of participating tasks and other problem parameters. Parametric studies

with this model identified the major contributors to parallelization penalty, and efforts were made to

reduce their effect on parallel speedup.

16

As a result the present multitasking algorithm incurs only

25-35%

of the parallelization penalty previously measured, and improvement in wall clock speedup

of up to 50% has been observed in some cases.

Graphics and Visualization

Earlier development of graphics capabilities for DORT utilized a commercial library to pro-

vide elementary graphics constructs. More recently ORNL has opted for public-domain-based

development (?f our neutronic codes graphics capabilities in order to facilitate distribution of the

package in a self-contained form for the users' convenience. For this purpose we adopted Sandia

National Laboratory's RSCORS graphics library which is distributed with future releases of

DOORS;

see Table 1.

The user

of

TORT

can

generate plots

of

flux

or

activities from formatted

or

unformatted files

generated

by the

code using

the

XTORID

and

ISPL3D codes.

17

Two

dimensional plots

are gen-

erated

for

selected planes

in the

TORT geometry,

and can be

viewed on-screen

for a

variety

of

plat-

forms,

or in

hardcopy. Several options

are

available

for

displaying

the

plotted quantity: color

and

grayscale shading, symbol

and

line contours.

In

addition,

the

geometric configuration

of the

selected plane

can be

overlaid

on the

plot.

Another ORNL graphics capability that

was

originally developed

for use

with

the

combina-

torial geometry models

of the

MASH shielding code

is the Oak

Ridge Geometry Analysis

and

Modeling Interface (ORIGAMI). ORIGAMI

is

XWindows based

and has

been developed

and

tested

on IBM

workstations.

Its

primary function verification

and

debugging

of

computational

models,

and

visualization

of

computed results. ORIGAMI

is not

distributed

in the

present release

of DOORS,

but

might

be

included

in the

future. Finally, users with specific interfacing needs

develop their

own pre- and

post-processors

for

TORT.

18

The seemingly endless increase

in

computational power,

i.e.

speed

and

storage size,

has

caused applications

to

grow

in

size resulting

in

extremely large data sets that

are

becoming harder

to manage.

The

need

for an

interactive Graphical User Interface

(GUI) is

evident

and we

hope

to

begin constructing

one for

DOORS soon.

References

1.

W. A.

Rhoades

and

Yousry

Y.

Azmy, "Three-dimensional

SN

Calculations with

the Oak

Ridge TORT Code," Proc.

Int. Conf. on

Mathematics

and

Computations, Reactor Physics,

and Environmental Analyses Portland, Oregon, April

30 - May 4, 1995, Vol. 1, p. 480,

Amer-

ican Nuclear Society, LaGrange Park,

IL

(1995).

2.

Y. Y.

Azmy, "Recent Advances

in

Neutral Particle Transport Methods

and

Codes,"

to

appear

in Proc. Fifth

Int. Conf. on

Applications

of

Nuclear Techniques, Crete, Greece, June

9-15,

1996.

3.

W. A.

Rhoades

and D. B.

Simpson,

"The

TORT Three-Dimensional Discrete Ordinates

Neutron/Photon Transport Code," ORNL/TM-13221,

to be

published.

4.

D. W.

Nigg,

et. al.,

"Demonstration

of

three-dimensional deterministic radiation transport

theory dose distribution analysis

for

boron neutron capture therapy," Medical Physics,

18, p.

43,

1991.

5.

R. A.

Lillie, "BNCT Filter Optimization Using

Two- and

Three-Dimensional Ordinates," these

proceedings.

6. Clifton

R.

Drumm, "Multidimensional electron-photon transport with standard discrete ordi-

nates codes," Proc.

Am.

Nucl.

Soc.

Topical

Mtg. on

Radiation Protection

and

Shielding,

p.

398,

No.

Falmouth,

MA,

April 21-25,

1996.

7.

Sedat Goluoglu

and Lee

Dodds,

"A

Deterministic Method

for

Transient, Three-Dimensional

Neutron Transport,"

in

preparation.

8. A. J. J. Bos, J. E. Hoogenboom, T. M. John, P. F. A. de Leege, "Application of TORT to a

Complicated N-Gamma Shielding Problem," these proceedings.

9. E. Botta, J. R. Galando, P. Neuhold, G. Saiu, "Three Dimensional Reactor Pressure Vessel

Fast Neutron Fluence Calculations for the AP600 using TORT," these proceedings.

10.

D. B. Simpson, "Splicing and Bootstrapping Methods for Coupling Primary DORT/TORT

Models to Secondary TORT Models," these proceedings.

11.

T. J. Burns, "A Hybrid Technique for 3-Dimensional Discrete Ordinates Analysis of Combina-

torial Geometry Models," these proceedings.

12.

W. F. Walters, "Augmented weighted-diamond form of the linear-nodal scheme for Cartesian

coordinate systems," Nuclear Science and Engineering, 92, pp. 192-196, 1986.

13.

R. L. Childs and W. A. Rhoades, "Theoretical basis of the linear nodal and linear characteris-

tic methods in the TORT computer code," ORNL/TM-12246, 1993.

14.

R. L. Childs, "Numerical methods for the flux solution," Workshop for the DORT and TORT

Radiation Transport Codes, No. Falmouth, MA, April 21, 1996.

15.

Y. Y. Azmy, "Analysis and Performance of Adjacent-Cell Preconditioners for Accelerating

Multidimensional Transport Calculations," these proceedings.

16.

Y. Y. Azmy, "Recent Improvements in the Performance of the Multitasked TORT on Time-

shared Cray Computers," these proceedings.

17.

C. O. Slater, "The XTORID and ISPL3D codes for plotting TORT activities," ORNL Internal

Memo,

March 25, 1996.

18

V

J. E. Hoogeriboom, T. M. John, A. Hersman, P. F. A. de Leege, "Pre- and Post Processing of

TORT Data and Preliminary Experience with TORT Version 3," these proceedings.