Content uploaded by Amit Acharya
Author content
All content in this area was uploaded by Amit Acharya on Jun 05, 2014
Content may be subject to copyright.
AIAA-2000-3456
1
American Institute of Aeronautics and Astronautics
Copyright © 2000 by the American
Institute of Aeronautics and Astronautics
I. D. Parsons, P. Alavilli, A. Namazifard,
A. Acharya, X. Jiao and R. Fiedler
Center for the Simulation of
Advanced Rockets
University of Illinois, Urbana, IL
Abstract
We describe simulations of solid rocket motors that
involve coupling between the core fluid flow, the
structural response of the propellant and case, and the
combustion of the propellant. A partitioned predictor-
corrector algorithm is employed to treat the fluid-
structure interaction. The combustion rate of the
propellant is coupled to the fluid flow via an empirical
power law relationship. Our algorithm couples the
physical processes involved using a partitioned
approach, enabling us to use existing codes to perform
the bulk of our simulations. We give special
consideration to the jump conditions that hold at the
fluid-structure-combustion interface, and specialize them
for the early burn phase. The interface between the
eroding solid and the fluid is treated using an ALE
formulation, which provides a consistent technique for
handling the eroding solid. Data are presented that
demonstrate the parallel performance of our code on a
variety of architectures. Results from simulations of the
space shuttle solid rocket motor demonstrate the
applicability of our approach. Future extensions of the
simulation capability to include thermal effects,
turbulence and material failure will be discussed.
1. Introduction
Simulation of solid rocket motors presents challenges
in many areas. A complete simulation requires
consideration of different physics (e.g., fluid flow,
structural deformation, fuel combustion, etc.), careful
treatment of the interface between these zones and
efficient parallel numerical algorithms. In this paper, we
outline a preliminary implementation of a parallel code
that couples the structural deformations experienced by
the rocket fuel and casing with the fluid flow in the core
of the rocket.
A code that will perform coupled multi-physics
simulations can be designed using either a monolithic or
a partitioned strategy. The monolithic approach requires
that a single new code be developed by the various
researchers who work in the different physics groups.
Global iterations are performed that simultaneously
update all of the variables in the different physics zones.
The interface conditions are enforced as part of the
global iterations. In contrast, the partitioned approach
uses existing application codes that may have been
developed by the different groups independently of any
integration effort. Interface conditions are enforced by
iteration between these different physics modules. Thus,
the partitioned approach provides a relatively
straightforward path for integration of the physics
demanded by a complete simulation of a solid rocket
motor.
The paper is constructed as follows. We first describe
the three component codes: ROCFLO, ROCSOLID and
ROCFACE. We then discuss the implementation and
parallel performance of a partitioned algorithm that
requires iteration to self-consistency between
ROCSOLID and ROCFACE. We end the paper with a
brief description of a large-scale demonstration
simulation of the space shuttle solid rocket motor.
2. ROCFLO - The Fluids Solver
ROCFLO is a CFD code developed to simulate solid
rocket booster core flow dynamics. Rocket flow
problems exhibit singular features compared to other
flow environments, primarily the fast propagating
acoustic pulses in the chamber that could potentially
cause instabilities and lead to rocket malfunction.
A three dimensional, structured, finite volume cell-
centered approach is adopted. In this framework
complex geometries are handled through a multiblock
approach. The multiblock approach also lends itself to
parallel computing. The Navier-Stokes equations are
solved on dynamic meshes whose boundaries adapt to
conform to the propellant surface that deforms due to the
loads imposed on it. Some details of the code are
presented in this section; for further details consult [1].
2.1 Governing Equations
The unsteady, three dimensional, compressible
Navier-Stokes equations on dynamic meshes may be
expressed as
..
t
¶
+Ñ=Ñ+
¶
U
FVS
(1)
where
U
is the set of conserved quantities,
F
is the
convective flux vector,
V
are the viscous fluxes and
S
are the source terms, i.e,
COUPLED SIMULATIONS OF SOLID
ROCKET MOTORS
2
American Institute of Aeronautics and Astronautics
11
1
222
33
3
1
1
2
2
3
3
()
()
,,
()
()
()
0
,.
ii
iii
iii
iii
iii
i
i
i
i
ijij
ug
ugupu
ugup
u
ugupu
Eugpu
E
u
u
u
qu
E
ρ
ρ
ρδρ
ρδ
ρ
ρδρ
ρ
ρ
ρ
τ
ρ
τ
ρ
τ
ρ
τ
ρ
éù
éù
-
êú
êú
êú
êú
-+
êú
êú
êú
êú
== -+
êú
êú
êú
êú
-+
êú
êú
êú
êú
-+
êú
êú
ëû
ëû
éù
é
êú
ê
êú
ê
êú
ê
êú
ê
==-Ñ
êú
ê
êú
ê
êú
ê
êú
ê
+
êú
ë
ëû
UF
VSg .
ù
ú
ú
ú
ú
ú
ú
ú
ú
êú
û
(2)
Here,
,,,,
i
upE
ρ
g
are the density, velocity
components, pressure, energy and grid speeds,
respectively. The equations
( )
222
123
12
p
Euuu
ρ
ρ
γ
=+++
-
(3)
and
pRT
ρ
=
(4)
complete the system. The stress tensor
ij
τ
and heat
flux
i
q
are given by
2
3
j
ik
ijij
jik
u
uu
xxx
τµµδ
æö
¶
¶¶
÷
ç
÷
ç
=-++
÷
ç
÷
ç
÷¶¶¶
ç
÷
èø
(5)
and
i
i
T
q
x
η
¶
=
¶
. (6)
2.2 Numerical Method of Solution
The above set of differential laws, governing the fluid
motion of a calorically perfect gas, may be written in the
integral form
( )
.
dVUdAdV
t
¶
+--=
¶
òòò
UFVgnW .
(7)
Over a computational cell (control volume) the semi-
discrete form of the previous equations becomes
( ) ( )
.
faces
d
UUAW
dt
Ω∆Ω
+-=
å
gnF (8)
where
F
is the numerical flux function,
Ω
is the cell
volume and
A∆
are its surface areas. The correction to
the fluxes due to mesh movement is .
UA
∆
gn
. In the
coupled rocket simulations, the grid speeds arise due to
the boundary deformations induced by the fluid loads
and as computed by the structures code. For dynamic
mesh computations the geometric conservation law is
another constraint that is satisfied in the computations.
The spatial discretization schemes implemented in
ROCFLO are the central scheme with artificial
dissipation [5] and two second order TVD schemes:
Roe's flux difference splitting scheme [11] and Yee's
symmetric TVD scheme [12]. A number of limiters, i.e.,
minmod, van Leer, van Albada and Superbee, have also
been implemented. Yee's symmetric TVD scheme, along
with one of the more diffusive limiters such as minmod,
have been shown in the literature to perform very well in
rocket chamber type simulations. Consequently this
scheme is generally utilized in computations.
The flow equations are marched in time using an
explicit multistage Runge-Kutta method. For the
unsteady computations of the integrated rocket code, a
two stage method is used as well as global time stepping,
where the marching step size is limited by the smallest
time step of all the cells in the computational domain.
2.3 Parallel Code Performance
ROCFLO is implemented in Fortran90. Fortran90
offers many features useful for parallel programming
over FORTRAN77. Primary of these are the user defined
data types. A computational block may be defined as an
object and contains all data relevant to that block (node
coordinates, face normals, volumes, solutions,
parameters, etc.). This object model facilitates placement
and migration of computational blocks on processors.
The code was initially optimized for scalar performance
both with respect to minimization of operations and also
cache utilization. The scalar performance of the code is
summarized in Table 1. The code executes at about 80
Mflops on a single processor of a Origin2000.
Scheme
central scheme
central + moving grids
2nd upwind scheme
2nd upwind + moving grids
s/node/RK-stg
7
12
10
15
Table 1: Scalar performance of ROCFLO.
The parallel scalability of ROCFLO on several
architectures is depicted in Figure 1. The test problem is
scaled such that as the number of processors is increased,
the total work load for each processor is constant. The
code scales very well on both of the machines. Further
refinements are in progress.
3
American Institute of Aeronautics and Astronautics
Figure 1: ROCFLO parallel scalability.
3. ROCSOLID - The Structures Solver
ROCSOLID, the structural analysis code used in the
coupled simulations, employs a finite element
discretization of the problem domain using unstructured
meshes. Dynamic problems are solved using the implicit
Newmark time integrator [2]. The linear matrix
equations encountered within the Newton iterations at
each time step are solved using a scalable parallel
multigrid solver [8]. The code is written in Fortran90,
and uses MPI to perform interprocessor
communications.
Kp
nonblocking MPI
Figure 2: Communications during Kp calculations.
Examination of the multigrid algorithm demonstrates
that all of the operations can be performed independently
on partitioned domains [10]. In particular, the main
components of the algorithm are matrix-vector
multiplications that can be efficiently implemented
element-by-element. Interprocessor communications are
only required during the matrix-vector multiplications,
scalar products and fine-to-coarse mesh restriction.
Matrix free element computations reduce the storage and
the time requirements of our implementation; for
example, the product
Kp
, where
K
is a stiffness matrix
and
p
is the search direction in conjugate gradient
iteration, can be written as
e
eeTe
ee
dV
æö
÷
ç
÷
ç
÷
ç
==
÷
ç
÷
ç
÷
÷
ç
ç
÷
èø
åå
ò
KpKpBDBp
R
. (9)
Computations can then proceed from right to left.
10
100
1000
10000
100000
1 10 100 1000
SGI O2K
IBM SP2
CRAY T3E
Number of Processors
Mflops
Figure 3: ROCSOLID parallel performance.
Figure 2 shows the communications required by our
distributed memory implementation of the matrix-vector
products. The mesh is partitioned into a number of
domains with equal numbers of elements. The product
Kp
is formed locally on each processor using the
approach outlined in Equation (9); interprocessor
communications are then performed using nonblocking
MPI communications as shown in Figure 2.
Multigrid methods require a hierarchy of increasingly
finer meshes. We use Truegrid to produce a sequence of
nested, uniformly refined hexahedral meshes, which
allows us to model complex parts. Mesh partitioning is
performed on the coarsest mesh using Metis to achieve
perfect load balance between the processors. Uniform
refinement of the coarsest mesh partitions produces the
required partitions on all of the fine meshes. Thus,
perfect element load balance is maintained through the
mesh hierarchy, although the resulting communication
pattern may not be optimum.
Figures 3 and 4 show data that measure the parallel
performance of ROCSOLID. Both sets of data were
obtained by solving a series of scaled problems on three
different parallel computers: a 512 processor Cray T3E,
a 128 processor Origin 2000 and a 64 processor IBM
SP2. The scaled problems were constructed so that the
amount of work for each processor (as measured by the
number of elements assigned to each processor)
remained constant. Figure 3 shows the measured
performance on each of the machines. Although the
4
American Institute of Aeronautics and Astronautics
observed Mflop rates are below the maximum quoted for
the different machines, the observed values are
reasonable for an unstructured mesh solver. The scaled
speed-ups shown in Figure 4 indicate the excellent
parallel performance of ROCSOLID. In particular, the
code runs extremely well on the Cray T3E. The poorer
performance on the Origin is probably attributable to
system processes invading the dedication partition used
to collect these data.
Number of Processors
Speed-Up
1
10
100
1000
1 10 100
1000
SGI O2K
IBM SP2
CRAY T3E
Ideal
Figure 4: ROCSOLID scaled speed-ups.
4. ROCFACE - The Interface Code
The interface code handles mesh association and data
transfer between the fluid and solid meshes across
multiple processors. The mesh association algorithm is
used to determine the geometric relationship between the
two non-matching meshes by locating the closest solid
element for each fluid nodal point on the interface. Our
algorithm traverses the fluid nodes and solid elements
from neighbor to neighbor so that the closest elements
can be located quickly [6]. We perform mesh association
at every time step to track the moving interface. Since
the geometric relationship changes rather slowly
between the two meshes, from the second time step we
search for the closest element of a fluid node starting
from its closest element in the previous time step to
speed up the search.
The result of the mesh association is then used for
motion and load transfer between the fluid and solid
meshes. It is essential to ensure conservation during data
transfer; we are currently using the methods in [4].
Specifically, we transfer displacements and velocities
from solids to fluids by interpolating the values for each
fluid nodal point at its closest point on the solid mesh.
For load transfer, the nodal forces for each solid node is
evaluated as a weighed sum of the fluid nodal forces.
The method uses the same set of coefficients for both
load and motion transfer, and as a consequence,
guarantees global conservation of energy [4].
Since the fluid and solid meshes are distributed across
multiple processors, the interface code must handle
distributed meshes. In our parallel implementation, we
first compute the bounding boxes of the partitions of the
solid mesh and of the blocks of the fluid mesh. These
bounding boxes are compared to provide a quick
estimate of the geometric relationship between the solid
partitions and the fluid blocks. Then the solid partitions
adjacent to a fluid block are shipped to the processor that
owns the fluid block, and are connected together to form
a new mesh. The sequential mesh association algorithm
is then applied on each processor in parallel. As the by-
product of mesh association, a more accurate
communication pattern is obtained which is used to
communicate the physical quantities between processors
to perform data transfer. The interface code is
implemented in C++ using object-oriented techniques,
employing the CGAL library (www.cs.uu.nl/CGAL) for
the geometric primitives and the half-edge data structure
for unstructured meshes.
5. GEN1 - A Multiphysics Code for Rocket
Simulations
The three component codes described above form the
basis of our rocket motor simulation tool. The key to
successfully solving our coupled problem is to capture
the interface physics correctly. Figure 5 shows the
combustion interface between the solid and fluid
domains; the interface moves with velocity
r
&
, and has
an outward unit normal
n
measured positive into the
solid domain. Consideration of conservation of mass and
linear momentum requires that
(
)
(
)
....
ssff
m
ρρ
-=-=
rnvnrnvn
&&
(10)
and
(
)
(
)
()()
fssf
m
-+-=
nn
vvtt0
(11)
respectively. Subscripts
f
and
s
denote the fluid and
solid regions, respectively,
ρ
denotes density,
m
denotes the mass transfer into the fluid,
v
denotes
fluid
solid
Figure 5: Interface geometry.
5
American Institute of Aeronautics and Astronautics
velocity and
()
n
t
denotes tractions measured with
respect to
n
. In order to model solid rockets, we make
some simplifying assumptions regarding the motion of
the fluid-solid interface.
5.1 Stationary Interface
We first consider a stationary interface, where the
interface is constrained to move with the solid, i.e., the
interface regression rate,
r
&
, is equal to the solid
velocity,
s
v
. This assumption allows us to model the
initial burn of the motor (e.g.,
1t
<
sec), during which
the regression of the interface through the solid is small.
In order to model the effect of the mass transfer from the
propellant to the core flow, we allow the interface to
transfer mass at a rate per unit area
m
, where
n
sf
map
ρ
=
, (12)
which is a standard combustion law. Consideration of
the balance of mass and linear momentum produces
..
fs
f
m
ρ
=-+
vnvn
(13)
and
sff
pm
=-
tnv
, (14)
respectively.
5.2 Moving Interface
In a simulation of the full rocket burn, we must
consider a moving interface. Here, the interface will
move through the solid according to the chosen
combustion law. Following the standard combustion law
used for the stationary interface, we have
(
)
.
n
sf
ap
-=
rvn
&
. (15)
The no-slip fluid boundary condition requires that
(
)
.
t
fss
=-
vvvnn
(16)
and balance of mass and linear momentum require
( )
..
n
n
s
fs
f
ap
ap
ρ
ρ
=-++
vnvn
(17)
and
(
)
()() n
sfsfsf
apρ
-
=-+-
nn
ttvv
, (18)
respectively.
5.3 Predictor-Corrector Coupling Algorithms
The interface conditions presented above are enforced
using predictor-corrector cycles that iterate between the
fluid and solid discretizations until self-consistency is
1n
t
n
t
f
t
s
t
11
,
nn
tt
ff
p
++
v
11
,
nn
tt
ss
++
dv
11
,
nn
tt
ff
++
dv
1n
t
s
+
t
n
t
Figure 6: Stationary interface predictor-corrector
algorithm.
obtained within a global time step. The algorithms
follow methods described in the aeroelasticity literature
[3,7], and are outlined in Figures 6 and 7 for the
stationary and moving interfaces, respectively. To
advance the solution from time
n
t
to
1
n
t
+
by advancing
over a global time step
n
t
∆
, we first advance the fluid
solution a number of explicit time steps (e.g., 10). This
produces estimates of the fluid interface pressure and
velocity at time
1
n
t
+
,
1
n
t
f
p
+
,
1
n
t
f
+
v
, respectively. Using
Equations (6) and (10), the tractions acting on the solid
at the interface,
1
n
t
s
+
t
, are computed. These tractions are
used as applied loads over an implicit solid time step
(
sn
tt
∆∆
=
). For coupling purposes, in the case of the
stationary interface the solids calculation produces
estimates of the interface position and velocity at
1
n
t
+
,
1n
t
n
t
f
t
s
t
n
t
1n
t
f
+
v
1
,
n
t
s
+
rt
1
n
t
s
+
v
11
,,
nn
tt
ff
++
rtv
Figure 7: Moving interface predictor-corrector
algorithm.
6
American Institute of Aeronautics and Astronautics
1
n
t
s
+
d
and
1
n
t
s
+
v
; in the case of the regressing interface, the
solid calculation generates the solid velocity at the
interface
1
n
t
s
+
v
. Using Equations (5) and (9), these
quantities provide the fluid boundary conditions that can
be used to drive the explicit fluids simulation, if
necessary. In the case of the stationary interface they
also prescribe the motion of the moving boundary for the
fluid calculation; otherwise Equation (7) is used.
The procedures shown in Figures 6 and 7 are repeated
until the interface quantities have converged. We
examine the relative change of the nodal velocities,
displacements and forces on the solids side of the
interface. When these quantities are small, we assume
that the interface has converged. The algorithms used in
ROCFACE ensure that conservative operators are used
to transfer quantities across the fluids-solids interface.
0
50
100
150
0 50
100
150
ideal
NCSA O2K
Number of Processors
Speed-Up
Figure 8: GEN1 scaled speed-up.
6. GEN1 Parallel Performance
We have already demonstrated the scalable parallel
performance of the primary GEN1 components codes
ROCFLO and ROCSOLID. The partitioned approach
adopted for the coupling algorithms preserves this
perfomance for GEN1. This is demonstrated in Figure 8,
which shows the scaled speed-up of the coupled code
measured on a 128 processor Origin 2000. Good
scalability is observed, although a reduction in
performance is observed when all 128 processors are
used.
Figure 9 shows the individual time requirements of
ROCSOLID, ROCFLO and ROCFACE measured during
the scaled speed-up runs. Clearly, the time spent in
transferring data across the interface is small compared
to the computation time in the two physics codes. With
our current strategy, twice as much time is spent in
ROCSOLID than in ROCLFO. However, this is partly
due to the use of single precision arithmetic in ROCFLO
and double precision arithmetic in ROCSOLID. Current
research is being directed at investigating the trade-offs
Number of Processors
Time (secs)
0
10
20
30
40
50
60
70
116 32 64 128
Interface
Fluids
Solids
Figure 9: GEN1 division of labor on the Origin 2000.
between the time steps used in the solid and fluid
simulations, accuracy and stability of the coupling
algorithms and computational cost.
6. Space Shuttle Solid Rocket Motor Simulations
Our code is continuing to be developed to simulate a
variety of solid rocket events. As a demonstration of our
capabilities, we recently used a 256 processor
Origin2000 in dedicated mode to simulate the first 0.1
secs of the firing of the space shuttle’s solid rocket
booster. ROCFLO used a discretization of 4,000,000
cells and ROCSOLID used a mesh of 270,000 elements
(the fluid cells were typically ten times smaller in linear
dimension that the solid elements). The fluids time step
was
6
10
-
secs., resulting in a global and solids time step
of
5
10
-
secs. Further details and visualizations produced
from this simulation are available at www.csar.uiuc.edu.
Currently, our coupled simulation capability is focused
on relatively simple models of the various physical
effects in the rocket. In the future, we will augment our
basic models to include features such as thermal effects
in the propellant and case, turbulence on the core flow
and material failure in the grain. Our choice of a
partitioned coupling strategy makes these formidable
tasks possible: experts in these different fields are able to
implement their models within the basic framework
outlined in this paper, using predictor-corrector iterations
to resolve conditions at the combustion interface.
References
[1] Alavilli, P. V. S., Tafti, D. and Najjar, F. 2000. The
development of an advanced solid rocket flow
simulation program ROCFLO. AIAA Paper 2000-
0824, 38
th
AIAA Aerospace Sciences Meeting and
Exhibit, Reno, NV.
[2] Bathe, K. J., 1996. Finite element procedures.
Prentice-Hall.
7
American Institute of Aeronautics and Astronautics
[3] Farhat, C., Lesoinne, M. and Maman, N., 1995.
Mixed explicit/implicit time integration of coupled
aeroelastic problems: three-field formulation,
geometric conservation and distributed solution.
International Journal for Numerical Methods in
Fluids, 21, 807-835.
[4] Farhat, C., Lesoinne, M. and LeTallec, P., 1998.
Load and motion transfer algorithms for
fluid/structure interaction problems with non-
mathcing discrete interfaces. Computer Methods in
Applied Mechanics and Engineering, 157, 95-114.
[5] Jameson, A., Schmidt, W. and Turkel, E., 1981.
Numerical solutions of the Euler equations by finite
volume methods using Runge-Kutta time stepping
schemes. AIAA paper 81-1259.
[6] Jiao, X., Edelsbrunner, H. and Heath, M. T., 1999.
Mesh association: formulation and algorithms. 8
th
International Meshing Roundtable, South Lake
Tahoe, 75-82.
[7] Lohner, R. et al., 1995. Fluid-structure interaction
using a loose coupling algorithm and adaptive
unstructured grid. Computational Fluid Dynamics
Review 1995, John Wiley, New York, 755-776.
[8] Namazifard, A. and Parsons, I. D., 1998. Parallel
multigrid methods for structural mechanics using
Fortran 90 and MPI. Poster presentation at
Supercomputing 98.
[9] Parsons, I. D., Alavilli, P. V. S., Namazifard A.,
Hales, J. and Tafti, D., 1999. Coupled multi-physics
simulations of solid rocket motors. Proceedings of
the PDPTA'99 International Conference, 3101-3107.
[10] Parsons, I. D., 1997. Parallel adaptive multigrid
methods for elasticity, plasticity and eigenvalue
problems. Parallel Solution Methods in
Computational Mechanics, Papadrakakis, M., editor,
Wiley, 143-180.
[11] Roe, P. L., 1981. Approximate Riemann solvers,
parameter vectors, and difference schemes. Journal
of Computational Physics, 40, 263.
[12] Yee, H. C., 1987. Construction of explicit and
implicit symmetric TVD schemes and their
applications. Journal of Computational Physics, 68,
151-179.
