Content uploaded by David Nordsletten
Author content
All content in this area was uploaded by David Nordsletten on Jun 28, 2017
Content may be subject to copyright.
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
SIAM J. SCI.COMPUT.c
⃝2016 Society for Industrial and Applied Mathematics
Vol . 38 , N o . 3, p p . C1 5 0– C1 7 8
MULTIPHYSICS COMPUTATIONAL MODELING IN CHeart∗
J. LEE†,A.COOKSON
†,I.ROY
†,E.KERFOOT
†,L.ASNER
†,G.VIGUERAS
†,
T. SOCHI†,S.DEPARIS
‡,C.MICHLER
†,N.P.SMITH
§,AND D. A. NORDSLETTEN¶
Abstract. Fr om ba s i c s c ien c e t o tran s l a t ion , mo de r n b i ome d i c al res e a r ch d e m a nds co m p uta-
tional models which integrate several interacting physical systems. This paper describes the infra-
structural framework for generic multiphysics integration implemented in the software CHeart,a
finite-element code for biomedical research. To generalize the coupling of physics systems, we intro-
duce a framework in which the geometric and operator relationships between the constituent systems
are rigorously defined. We then introduce the notion of topological interfaces and define specific
operators encompassing many common model coupling requirements. These interfaces enable the
evaluation of weak form integrals between mesh subregions of arbitrary finite-element bases’ orders,
type s, and spatial dimensions. Equation maps are intro duced wh ich pr ovide abstra ct representations
of the individual physics systems that can be automatically combined to permit a monolithic matrix
assembly. Flexible solution strategies for the resulting coupled systems are implemented, permitting
fine-tuning of solution updates during fixed point iterations, and subgrouping where several prob-
lems are being solved together. Partitioning of coupled mesh domains for optimal load balancing is
also supported, taking into account the per-processor cost of the entire coupled problem within the
graph problem. The demonstration of the p erformance is illustrated through important real-world
multiphysics problem s relevant to cardiac physi ology.
Key words. cardiac modeling, multiphysics, parallel computing, coupled problems, modeling
software
AMS subject classifications. 92-04, 92-08, 74F10, 74S05, 76M10, 68W10, 92C30
DOI. 10.1137/15M1014097
1. Introduction. Computational modeling of biophysical phenomena has be-
come a common approach for investigating problems that arise in medical sciences.
From subcellular processes [22, 86] to whole-organ function [65, 81], mathematical
models continue to demonstrate their capacity to predict physiology, provide added
insight, and address questions of both scientific and clinical interest. The demand
for solving computational models has led to the development of a variety of soft-
ware packages [82, 83, 43, 74, 59, 24, 52, 73, 47, 71, 10, 75]. The success of many
of these tools stems from their ability to offer efficient and scalable computation
for specific physics problems. These packages widely employ finite-element methods
∗Submitted to the journal’s Software and High-Performance Computing section March 27, 2015;
accepted for publication (in revised form) March 28, 2016; published electronically May 26, 2016.
This work was supported by the British Heart Foundation (NH/11/5/29058), the Engineering and
Physical Sciences Research Council (EP/G0075727/2), and the Europ ean Commission funded eu-
Heart project (FP7-ICT-2007-224495:euHeart). It was also supported by the Wellcome Trust-EPSRC
Centre of Excellence in Medical Engineering (WT 088641/Z/09/Z) and the NIHR Biomedical Re-
search Centre at Guy’s and St.Thomas’ NHS Foundation Trust and KCL. The views expressed are
those of the authors and not necessarily those of the NHS, the NIHR, or the DoH.
http: //www.siam.org/journal s/s isc/38-3/M101409.html
†Department of Biomedical Engineering, King’s College London, London, UK (jack.lee@kcl.ac.uk,
andrew.cookson@kcl.ac.uk, ishani.roy@kcl.ac.uk, eric.kerfoot@kcl.ac.uk, liya.asner@kcl.ac.uk,
guillermo.vigueras gonzalez@kcl.ac.uk, taha.sochi@kcl.ac.uk, christian.michler@kcl.ac.uk).
‡Mathematics Institute of Computational Science and Engineering, EPFL, Lausanne, Switzerland
(simone.deparis@epfl.ch).
§Department of Biomedical Engineering, King’s College London, London, UK, and Faculty of
Engineering, University of Auckland, Auckland, New Zealand (np.smith@auckland.ac.uk).
¶Corresponding author. Department of Biomedical Engineering, King’s College London, London,
UK (david.nordsletten@gmail.com).
C150
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C151
Fig. 1.1.An example of finite-eleme nt meshes used for mult iphysics simulations. Illustrated
is the coupling between overlapping domains (hexahedral (1) and tetrahedral meshes (5, 6) for solid
mechanics, monodomain, and poroelasticity problems) and over partially overlapping domains (en-
docardial (2) and cavity mesh (3) for fluid-structure interaction (FSI) problems and vascular mesh
(4) for the coronary flow model).
(FEMs) for numerical approximation of differential equations and have introduced a
range of techniques that span the computational spectrum. From support for low
order h-refined techniques [82, 83], to high order p-FEMs [43], to varied mixed meth-
ods [24, 10, 33] and h-adaptive techniques [73, 52], diverse approaches for problem
solving have been enabled. A number of software packages also utilize mesh parti-
tioning (such as ParMETIS [44], Scotch [72], or Zoltan [19]) and parallel linear solver
libraries (such as Trilinos [34] and PETSc [5, 4, 6]) for efficient multicore paralleliza-
tion. These strategies for scability are necessary to address the computational chal-
lenges presented by many mathematical models (e.g., hemodynamics [74, 43, 52, 24]
and electrophysiology [59, 82, 83, 73]).
Despite these advances, the seemingly simple task of multiphysics modeling by in-
tegrating existing computational models remains challenging. For a general coupling
between arbitrary problems, many aspects of the individual problem—including nu-
merical discretization and solution schemes, disparate solution domains, and data ac-
cessibility issues—have to be formalized, presenting new implementational challenges.
For instance, in the context of cardiac modeling a variety of numerical techniques are
employed (see Figure 1.1), spanning from the one dimensional (1D) network mani-
folds used in vascular flow modeling [79], to curvilinear high order elements used in
mechanics [61], to extremely refined tetrahedral meshes used in electrophysiology [62].
The resulting equations also require the ability to solve large scale linear systems as
well as highly nonlinear systems. The varied nature of these physical models leads to
many different forms of coupling, where integrated models may require equating kine-
matics/force (as found in fluid-solid interaction), mass transport/force (as found in
poromechanics or multilevel transport), or strain/tension (as found in electromechan-
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C152 LEE ET AL.
ics) between models on heterogeneous domains. The permutation of these factors
has made the generalized coupling of multiphysics systems a significant challenge to
the scientific community. While the integration of specific physical systems has been
explored in the literature [24, 52, 33], there is still much that can be gained from
a focused examination of the fundamental design that consolidates a multiphysics
solver.
In this paper we describe an infrastructural framework for generalizing the integra-
tion of physical phenomena within CHeart, a multiphysics software engine developed
at King’s College London. To date, it has been used to solve a variety of problems in
physiological modeling, including the Navier–Stokes equations for fluid flow [17], non-
linear solid and porous mechanics [13, 32, 31, 3, 2], scalar transport equations [60, 41],
elastic wave mechanics [49], pressure Poisson for approximating pressure from Navier–
Stokes using known velocities [48, 20], and electrophysiology [84]. These essential
single-physics building blocks provide the components for assembly and linking to
form multiphysics models. This is achieved through a software infrastructure using
what we call object-sets that act as generic sets to store different core FEM objects
that may be constructed, used, or shared by any multiphysics problem. This strat-
egy is used to manage bases, meshes and variables, topological interfaces, equation
abstraction through equation mappings, and a series of general physics couplings to
flexibly interlink heterogeneous physical models. These processes enable partitioned
coupling, monolithic coupling, or a mixed approach for different components of the
physical systems allowing for a multitude of possible solution strategies. The infra-
structural organization outlined has enabled the interlinking of a variety of models,
including fluid-solid coupling [67, 68, 70, 56, 18, 57, 58], vascular-porous flow [50, 51],
and electromechanics [84].
The remainder of the paper will describe a general strategy for multiphysics inte-
gration and demonstrate its application. In section 2 we outline the physics systems
which provide core simulation components for multiphysics integration. The general
object-set approach, construction of function and variable spaces as well as the intro-
duction of topological interfaces, is provided in section 3. A series of single-physics
coupling strategies is described in section 4. The generic construction and handling
of these different systems is introduced in section 5 with particular emphasis on their
abstraction through the use of equation mappings. In section 6, we propose a graph
parallelization strategy which enables efficient multicore processing. These techniques
are then validated and demonstrated on a series of multiphysics systems relevant to
cardiac physiology (see theonlinesupplement (M101409 01.pdf [local/web 4.23MB])),
with a specific example for fluid-structure interaction (FSI) provided in section 8.2.
2. Core single-physics module. In the finite-element framework the govern-
ing equations for physical systems are provided in standard weak form. In this section
we introduce a generic notation used for defining operators, or functionals, for dif-
ferent physical systems. These functionals provide the building blocks for integrated
multiphysics systems.
Different physical systems and their weak form functionals vary in the terms,
function spaces, number of variables, and their inclusion of different fields. Here we
consider each problem as a specialization of a general weak functional, f:Vh(Ωh)×
Yh(Ωh)→R,
(2.1)
f(vh,yh;Ω
h)=!Ωh
a(vh)·yh+c(vh):∇yhdΩh,vh∈Vh(Ωh),yh∈Yh(Ωh),
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C153
Tab l e 2 . 1
Summary of available options for different physics systems and physics-based operations.
Physics-based operators & operations
ODE/DAE systems Adaptive time stepping, implicit/explicit schemes (Euler, RK4, etc.)
Lumped parameter cardiovascular models (incl. [77, 85])
Var i ou s c e l l mo d e l s (i n c l. t e n Tu ss che r 2 0 06 [8 0 ] , Lu o – Ru d y 1 9 91 [5 3 ] )
Dynamic loading of precompiled cell models
Solid mechanics Incompressible, penalty, weak penalty, nearly incompressible formulations
Arbitrary mixed formulations: displacement, velocity, pressure
Arbitrary constitutive laws (isotropic, orthotropic, anisotropic laws)
Summable constitutive laws, i.e., σ(x)=!kwk(x)σk(x)
Scalar transport Static/transient, inclusion/exclusion of advection and/or reaction
Optional SUPG stabilization
Solve as linear or nonlinear system
Fluid mechanics Stokes and Navier–Stokes, static and transient
Eulerian and arbitrary Lagrangian–Eulerian (ALE) formulations
Conservative and noncon servative ALE formulations
General mixed formulations
SUPG formulation for all forms
1D fluid flow Static/transient linear/nonlinear formulations
Area-flow or area-velocity formulations
Riemann invariant or compatibility-based b oundary condition formulations
Simplification to Poiseuille network flow (compliant and rigid vessel wall)
Accommodates arbitrary wall constitutive mo dels
Primitive variable or Windkessel-type boundary conditions
Pressure Poisson Stokes and Navier–Stokes, static and transient
Elastic wave equation Linear and nonlinear formulations
Norms & measures Compute L2,H1norms/seminorms of variable expressions
Compute general inner-product expressions
Projections Compute L2,H1,andHdiv projection operators
Parameter e stim atio n Reduced-order unscented Kalman filter
Support for arbitrary spatially varying parameters
where vhis an n-dimensional variable (which may consist of one or more state vari-
able(s)), yhis the nth-dimensional test function, Vh(Ωh)andYh(Ωh)aretheset
of trial and test spaces, respectively, and Ωh⊂Rdis the FEM domain description.
Here,
a:Vh→"L2(Ωh)#n,c:Vh→"L2(Ωh)#n×d
are specific functionals of the variable, vh, which depend on the underlying physical
system. In a single-physics example, the functional fwould be used to find the
solution vhin the weak form statement: find vh∈Vh(Ωh)suchthat
(2.2) f(vh,yh;Ω
h)=0 ∀yh∈Yh(Ωh).
As the variability between physics arises predominantly in the aand cfunctionals, this
abstract definition of the weak form problem enables generic handling of the core FEM
procedures. In the remainder of this section we introduce the specific core physics
currently implemented in CHeart. A summary of these modules is also provided in
Table 2.1.
2.1. Scalar transport. The conduction of electrical current, movement of me-
tabolites, and conduction of simple flow potential problems are a few examples of
a wide group of transport processes. Their simulation can be achieved through the
solution of the transient scalar advection-diffusion-reaction equation or some simpli-
fication of it (see Table 2.1). As a result, scalar transport equations encompass a
broad range of problems ranging from the most simplified form, Laplace, to the full
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C154 LEE ET AL.
transport system. In the case of a transient advection-diffusion-reaction equation
discretized using the backward Euler scheme, the operators in (2.1) have the form
a(vh) := 1
δt
(vk
h−vk−1
h)+u·∇vk
h−λvk
h+s, c(vh) := D∇vk
h,
where vk
h∈Vh=Vhis the unknown scalar value at the kth time step, δtis the time
step interval, uis an advective velocity field, λis a linear reaction coefficient field, D
is a tensor diffusion coefficient field, and sis a source/sink field.
2.2. Solid mechanics. Solid mechanics in cardiac applications is solved us-
ing large strain finite deformation theory [55, 9, 7] within quasi-static or transient
frameworks. The specific formulations can vary depending on whether the material is
modeled as compressible, incompressible, or nearly incompressible [32] as well as the
specific selection of constitutive laws (see Table 2.1). As an example, a quasi-static
incompressible material can be modeled by (2.1) using the aand coperators
a(uh,p
h) := (ρb,det |∇uh+I|−1),c(uh,p
h) := (σ(uh)−phI,0),
where (uh,p
h)∈Uh×Wh=Vhare displacement and pressure, ρand bdenote mate-
rial density and body force, respectively, σ(uh) is the deviatoric Cauchy stress tensor
which may be defined as an arbitrary sum of core constitutive laws (see Table 2.1)
scaled by a weighting field, and Iis the identity tensor. Currently, a collection of
isotropic (i.e., neo-Hookean, Mooney–Rivlin [9]), general anisotropic constitutive laws
(i.e., Costa [14], Guccione [30], and Holzapfel–Ogden [36]) and viscoelastic models are
implemented and can be selectively applied tothetotalorthedeviatoriccomponent
of the strain tensor. Extensions to the existing constitutive laws are achieved by the
generalized implementation of the stress tensor, enabling straightforward inclusion of
new materials.
2.3. Fluid mechanics. Fluid mechanics can be solved for a variety of problems
in the heart (see Table 2.1), ranging from steady Stokes flow in small coronary vessels
to the more complex arbitrary Lagrangian–Eulerian (ALE) form of Navier–Stokes
equations [35, 40, 37, 64, 25, 66, 70] in the cardiac heart chambers. As an example, in
the nonconservative form of the Navier–Stokes equations discretized using backward
Euler, the operators aand cof (2.1) are written as
a(vh,p
h):=!ρ
δt
(vk
h−vk−1
h)+ρ(vk
h−wk
h)·∇vk
h,∇·vk
h",c(vh,p
h):=(µσ(vk
h)−pk
hI,0),
where (vk
h,p
k
h)∈Vh×Wh=Vhare the fluid velocity/pressure at the kth time step
with step-size δt;ρ, µ are the density and viscosity; and σ(vh) is the stress operator
(see Table 2.1). By using various bases, Vh×Whmay be defined for a wide range of
inf-sup stable spaces (see section 3.4), such as Pκ−Pκ−1Taylor–Hood elements [11],
Qκ−Pκ−1Nicolaides–Boland [8] elements, and Crouzeix–Raviart [16] spaces as well
as others. Time step handling, in general, is consistent with that shown in [70] for
ALE formulations. Further, as stabilization is often required for practical problems,
the test and trial spaces may bedifferent,enablinggeneralintroductionofstreamline
upwinding Petrov–Galerkin (SUPG) methods [38].
2.4. Darcy flow. Darcy’s law in combination with conservation of mass is used
to model the flow of a fluid through a permeable porous medium [15, 60, 13]. In the
steady form, the operators aand cof (2.1) are written as
a(vh,p
h) := (µK−1vh,∇·vh−s),c(vh,p
h) := (−ph,0),
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C155
where (vh,p
h)∈Vh×Wh=Vhare the Darcy velocity and pore pressure, respec-
tively, µis the dynamic viscosity, Kis the permeability tensor, and sis a volumetric
flow rate source or sink. Currently, this implementation relies on inf-sup stable spaces
Vhand Wh.
2.5. 1D fluid flow. The 1D fluid flow equations in an elastic vessel can be
obtained from the incompressible Navier–Stokes equations written in cylindrical co-
ordinates by assuming axisymmetric flow and by absorbing the dependence on the
radial coordinate into a lumped parameter [39, 26, 76]. This results in an equation
set defined by
a(Qh,A
h) := $Qk
h−Qk−1
h
δt
+κQk
h
Ak
h
,Ak
h−Ak−1
h
δt%,
c(Qh,A
h) := −$α(Qk
h)2
Ak
h
+!c2
wdA, Qk
h%,
where (Qk
h,A
k
h)∈Vh×Wh=Vhis the fluid volumetric flow rate/vessel cross-
sectional area at the kth time step with step-size δt,κis a friction coefficient, cw
is the local pulse wave velocity, and αis the momentum correction factor which
depends on the assumed fluid velocity profile. At junctions where branching occurs,
the above equations are further augmented by a set of coupling conditions, i.e., mass
conservation and the Bernoulli equation [76].
2.6. ODE and DAE systems. Various systems of ordinary differential equa-
tions (ODEs) and differential algebraic equations (DAEs) ranging from 0DWindkessel
models [85, 78, 1] to tissue-level cell models [23, 53, 80] can be described using the
equation v′=g(v), where gis known as the rate function. The solution of such prob-
lems can be obtained using explicit, semi-implicit, or implicit time integration with
constant time steps or adaptive time stepping. For example, the standard explicit
Euler scheme can be written in the form of (2.1) using the aand coperators,
a(vh) := 1
δk
t
(vk+1
h−vk
h)−g(vk
h),c(vh) := 0,
where vk
h∈Vh=Vhis the state variable, and δk
tis the step-size at the kth time
step. The domain of the problem is normally either a point or a series of points which
effectively decouples the solution to individual nodes or modes (i.e., for a system with
nvariables, vh∈[L∞(Ωh)]n).
In practice, directly incorporating a wide spectrum of ODE and DAE models into
the code base requires frequent editing and recompilation. Flexibility is achieved in
CHeart through dynamic loading of user-defined rate functions g.Thisallows,for
example, a direct C-code export from online model repositories such as CellML1to
be integrated without global recompilation.
3. FEM weak form construction. The core physics-based operators intro-
duced in the previous section rely on a series of routines common to all FEM codes.
The additional challenge of multiphysics comes with the efficient reuse, integration,
and cross-communication of fields, domains, and weak form functionals. Critical to
this is the formation of object-sets—representing a collection of core objects—that may
be reused freely. An illustrated example of how different object-sets within CHeart
interact together is shown in Figure 3.1.
1www.cellml.org
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C156 LEE ET AL.
VARIABLES PHYSICS SOLVER SOLUTION DOMAIN
Solver
matrix
(FSI)
Solver
matrix
(ALE)
Sequential
fixed-point
(line search)
Math
Expression
Section 3.5
Basis
Function
Section 3.1
Mesh
Topology
Section 3.2
Solver
Matrix
Section 5.3
Solve
Group
Section 5.5
Interface
Section 3.3
Equation
Section 2
Variable
Section 3.4
Identity
Linear fluid
domain
Identity
Cubic solid
domain
Quadratic
solid domain
Injective
Fluid pressure
Fluid velocity
Fluid Space
Domain velocity
Boundary Space
Lagrange multiplier
Solid displacement
Solid pressure
Navier-
Stokes
Solid
mechanics
ALE
problem
Coupling
problem
(
LM
)
Linear
tetrahedral
Quadratic
tetrahedral
Quadratic
triangular
Cubic
hexahedral
Quadratic
hexahedral
Quadratic
fluid domain
Boundary
surface
Fig. 3.1.Illustration relating various object-sets used by CHeart to solve physics problems.
In this case, we consider the coupled fluid-solid interaction (see Results). The fluid is solved on
atetrahedralgridusingP2−P1Ta ylo r–H ood e le m ent s a nd t he AL E N avi e r–S t ok e s for m ul a tio n
introduced in section 2.3.ThesolidissolvedonahexahedralcurvilineargridusingQ3−Q2Tayl or–
Hood elements and the solid mechanics formulation from section 2.2.Object-setBis constructed
using five basis object types with varying order shape and dimension. These are used with five
topology objects comprising T,andtheinterfaceIobject-set is composed of two identity and two
injective interfaces. Vis then built from eight variables used in the four core problems feeding into
the monolithic solver.
In this section we review the core routines implemented to facilitate multiphysics
integration. As bas is interpolation schemes may be widely different, we formulate a
basis object-set for which we developed a single strategy for computation and interpo-
lation of different basis functions. These basis tools form the foundation for the gen-
eration of topologies—meshes with specified interpolation—which are integrated into
a topology object-set forming flexible structures to house meshesofdifferent order,
size, dimension, etc. To enable cross-communication between fields and topologies, we
introduce a general construct of interfaces that allows the formulation of functionals
which relate variables defined on different topologies. With these tools, we construct
variable object-sets, which may be used in the general functionals introduced above
to assemble weak form statements.
3.1. Basis generation. The core physics modules presented in section 2 re-
quire a range of different interpolation functions that may rely on different refer-
ence elements, interpolation orders, and expansion types. This is handled by con-
struction of a basis object-set Bwith each individual basis object, B, being defined
as B={τ,{ϕ1,...,ϕ
K},Q}.Hereτdenotes the master (or reference) element,
{ϕ1,...,ϕ
K}is a set of K-basis functions used to construct finite-dimensional func-
tion spaces, and Q={(wp,ξp),p=1,...,N
p}defines the quadrature rule via the list
of weights wp∈R+and points ξp∈τ. The full list of available options is detailed in
Table 3.1. All bases defined for a sp ecific problem constitute the basis ob j e c t - s e t B.
A wide range of basis types, including modal [43], nodal [28, 88], Crouzeix–Raviart
[16], and bubble functions [28], can be written using the cardinal expansion. If Mis
the total number of unique polynomial terms of B,d=dim(τ), and Kis the number
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C157
Tab l e 3 . 1
Summary of available element/basis specifications in CHeart.Optionsareavailableinany
combinati on (ex cl ud in g poi nt e le me nt ty pes which cannot s up port i nt er pol at io n).
Element/basis specifications
Element types point, line, quadrilateral, triangle, tetrahedral, hexahedral, prism
Basis types nodal, continuous/discontinuous, order (0–5)
modal, continuous/discontinuous, order (1–15)
user specified‡
Quadrature schemes Gauss–Legendre, integrable order (0–60)
Keast–Lyness, integrable order (0–10)
‡:Userdefinedbasisenablesusageofalternativebasisdefinitions,includingbubbleandCrouzeix–
Raviart element types, which are supplied using a simple text file containing basis coefficients and
powers (i.e., αand β).
of functions in the basis B,thenϕi(i=1,...,K) and its derivatives can be computed
as
ϕi(ξ)=
M
&
j=1
α(i, j)
d
'
k=1
ξβ(k,j)
k,(3.1)
∂ϕi(ξ)
∂ξn1
1···∂ξnd
d
=
M
&
j=1
α(i, j)
d
'
k=1
D(βkj ,n
k)ξβ(k,j)−nk
k,
where ξ=(ξ1,...,ξ
d)Tis a local coordinate in τ,α∈RK×Mis the sparse coeffi-
cient matrix, β∈Rd×Mis the corresponding powers in the cardinal expansion, and
D(p, n)=(n−1
k=0 (p−k),D(p, 0) = 1. As shown in (3.1), the derivatives can be auto-
matically evaluated and refactored as a new sparse cardinal expansion with adjusted
coefficients. Here we note that for nodal Lagrange and modal basis functions the
number of terms in the cardinal expansion is equivalent to the number of basis func-
tions (i.e., K=M). However, in the case of bubble or Crouzeix–Raviart elements,
this is not the case.
The coefficient matrix αmay either be user-supplied or constructed algorithmi-
cally. For example, for nodal Lagrange polynomials, α=A−Tis the inverse transpose
of the Vandermonde matrix (i.e., A(i, j)=(d
k=1(ξi
k)β(k,j)with ξ1,...,ξK∈τbeing
nodal coordinates).
Furthermore, tensor-based expansions can be written in a similar form,
ϕi(ξ)=
d
'
k=1
K1/d
&
j=1
α(γ(i, k),j)ξβ(j)
k,
∂ϕi(ξ)
∂ξn1
1···∂ξnd
d
=
d
'
k=1
K1/d
&
j=1
α(γ(i, k),j)D(β(j),n
k)ξβ(j)−nk
k,
where α∈RK×K1/d and β∈RK1/d are the 1D sparse expansions, and γ∈RK×d
is an index array defining the corresponding 1D basis. For efficient computation the
expansion for each basis function is written as a sum of the nonzero components of
α.
Due to the arbitrary degree of the basis, a range of quadrature schemes is required.
Schemes specifically for triangular and tetrahedral elements, such as those by Lyness
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C158 LEE ET AL.
and Jespersen [54] and Keast [46], were integrated based on the source code provided
by [63]. Higher order Gauss–Legendre schemes are also available based on finding the
roots of the Jacobi polynomials [43], and using tensor product expansion/collapsed
expansions to integrate up to 60th order. In addition to these built-in bases, CHeart
supports arbitrary user-defined bases and/or quadrature schemes to be provided via
a simple text file.
3.2. Topologies. Similar to the outlined basis object-set B,anobject-setof
topologies Tis constructed. A topology object T∈Tin CHeart is constructed
using a specific basis B∈Band provides the mapping between mesh element indices
and global basis function (node or mode) indices used to evaluate fields over elements.
Specifically, the topology T={E,B,N
E,N
N}is comprised of NEand NN, denoting
the total number of elements and global basis functions, respectively, Bbeing the
basis used to perform field evaluations, and E∈NNE×Kthe element mapping such
that N=E(e, k)forsome e∈{1,...,N
E},k∈{1,...,K},N∈{1,...,N
N}.
3.3. Interfaces. By default, each topology T∈Tis defined as a singular entity
with an unknown relation to others in the object-set. However, in many cases, such
as mixed formulations or multiphysics interactions, it is desirable for a well-defined
relationship to exist between topologies, enabling the transfer of data from one to the
other. This may be accomplished by defining an interface object Ii→j∈I(with I
being the interface object-set) which effectively maps element indices and local element
coordinates from Tito Tj,Ti,T
j∈T. Each interface object Ii→j={Ti,T
j,ei,j ,Ii,j }
contains components to construct a mapping. Letting NE,i and NE,j be the numbers
of elements, and τiand τjthe master elements in Tiand Tj, respectively, then ei,j ∈
NNE,i defines the relationship between element indices in the two topologies, and
Ii,j ∈RNE,i×Kthe relationship between local coordinates. The mapping can be fully
described by the mapping ζi→j:{1,...,N
E,i}×τi→{1,...,N
E,j}×τj,
(3.2) ζi→j(e, ξ)=)ei,j (e),
K
&
m=1
Ii,j (e, m)ϕi,m(ξ)*=(e′,ξ′),
for some (e′,ξ′)∈{1,...,N
E,j}×τjand (e, ξ)∈{1,...,N
E,i}×τi.Themapping
ζi→jenables interpolation and evaluation of fields and field derivatives as will be
discussed in section 3.4.
In CHeart all interfaces are assumed to be null mappings by default, i.e., no
mapping exists between topologies. Various kinds of interfaces may, subsequently, be
defined. These are outlined below and illustrated in Figure 3.2.
Identity interface. The most commonly used interface in FEMs is the identity
interface, given in Definition 3.1 and shown in Figure 3.2a, which simply returns the
same element and local element coordinates. Here, the associated basis of each topol-
ogy must be compatible in that their respective master elements must be geometrically
equivalent and their number of elements identical.
Definition 3.1 (identity interface). Let Ti,T
j∈Twith NE,i =NE,j =NE
and τi=τj=τ.AninterfaceIi→j∈Iis called an identity interface if for any
(e, ξ)∈{1,...,N
E}×τthe mapping function ζi→j(e, ξ)=(e, ξ),whereζi→jis
defined by (3.2).
This is often employed for classic mixed-element formulations, such as those used
in fluid mechanics or solid mechanics formulations, where velocity/displacement share
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C159
Fig. 3.2.Types o f i nt e r fa ce s s up po rted in CHeart.(a) Identity interfaces between linear,
quadratic, and cubic quadrilateral meshes. (b) Injective interface. (c) Injective interface via an
intermediary. (d) An injectiv e interface between nonconforming meshe s through a conforming in-
termediary. (e) Adegenerateinterfacebetweena1Dvesselterminalandtheboundaryfaceofthe
cylinder.
the same mesh as pressure variables but use different interpolations (i.e., different
topologies).
Topology groups. As topologies linked by an identity interface share many
commonalities, it is often useful to define topology groups (see Definition 3.2), which
link these objects together. This is particularly convenient, as more complex interfaces
may then be defined for one member of the topology group and used for all other group
topologies.
Definition 3.2 (topology group). Let Tg⊆T;thenTgis a topology group if
there exists an identity interface Ii→j∈Ifor every Ti,Tj∈T
g.Further,Tg(i)is
referred to as the topology group of Tiif Ti∈T
g(i).
Injective interface. In multiphysics simulations the interconnectivity of topolo-
gies is often more complex. For this injective interface, mappings may be introduced
which provide a continuous map between element/local coordinate data and topolo-
gies. This may be on a portion of a domain, as in FSI where exchange between the
fluid and solid occurs on an interface. Definition 3.3 specifies the details of injective
interfaces which are also illustrated graphically in Figure 3.2b–d.
Definition 3.3 (injective interface). Let Ti,T
j∈T.AninterfaceIi→j∈Iis
cal led injective if for every (e, ξ)∈{1,...,N
E,i}×τithere exists a unique (e′,ξ′)∈
{1,...,N
E,j}×τjsuch that ζi→j(e, ξ)=(e′,ξ′),whereζi→jis defined by (3.2).
This type of interface provides a mapping for all element/local coordinate pairings
from Tiinto some or all pairings from Tj. As a result, the injective interface allows for
the embedding of refined grids or lesser dimensional manifolds into another topology.
This formulation enables a wide range of associations between topologies. Injective
maps can be used to couple subsets of two topologies where only a portion of each is
interfaced by introducing an intermediary topology which is injective to both over the
desired subsets (see Figure 3.2c). This need arises in fluid-solid coupling, for example,
where the interaction between fluid and solid domains occurs on some portion of the
domain boundaries (see section 8.2). Further, injective interfaces can be easily ex-
tended to fully nonconforming grids by introducing an additional polygonal topology
nested within the intersections of the interfacing topologies (see Figure 3.2d).
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C160 LEE ET AL.
An important property of injective interfaces is their transferability to the associ-
ated topology groups, which facilitates data transfer between single-physics problems
employing mixed interpolation spaces. As shown in Lemma 3.4, it is clear that an
injective interface Ii→j={Ti,T
j,ei,j ,Ii,j }can be extended to any Tk∈T
g(Ti) using
the mapping ζi→j.
Lemma 3.4. Suppose Ti,T
k∈T
g(i)and Ii→j∈Iis injective. Then, there is an
implicit mapping Ik→j∈Ithat is injective.
Proof. Since Ti,T
k∈T
g(i), then ζk→i(e, ξ)=(e, ξ)andIk→iis an identity in-
terface. The mapping for Ik→jcan therefore be constructed as ζk→j(e, ξ)=ζi→j◦
ζk→i(e, ξ)=ζi→j(e, ξ) for all (e, ξ)∈{1,...,N
Ek}×τk. Since the interface Ii→jis
an injective interface, Ik→jis also an injective interface.
An important subset of identity and injective interfaces includes those which
are complete (see Definition 3.5). By definition, all identity interfaces are complete.
However, in general, an injective interface enables linking one topology Tito a subset
of another Tj.Acompleteinterfaceensuresthatthissubsetextendstotheentirety
of Tj. This is particularly important for FEM weak form statements which must be
held over a defined domain which may require interfaces to other variables.
Definition 3.5 (complete interface). An injective interface Ii→j∈Iis called
complet e if ζi→jis bijective.
If an injective interface Ii→j∈Iis defined between two topologies Ti,T
j∈T,then
an inverse mapping can be implicitly evaluated and constructed (see Definition 3.6).
However, explicitly constructing Ij→iin the form shown in (3.2) is not, in general,
straightforward. Instead, the inverse of an injective interface is computed procedurally
by CHeart based on the provided injective interface.
Definition 3.6 (inverse interface). For an inj ect i v e interfa ce Ii→j∈Iwe may
implicitly define an inverse interface Ij→ithat for some (e′,ξ′)∈{1,...,N
E,j}×τj
there exists a unique (e, ξ)∈{1,...,N
E,i}×τisuch that ζi→j(e, ξ)−(e′,ξ′)=(0,0).
Degenerate interface. In certain cases, the requirements of injective interfaces
or their inverses can be too restrictive, requiring the construction of degenerate inter-
faces (see Definition 3.7), where a topology may be mapped nonuniquely. This case
arises when reduced-dimensional formulations are coupled to full-dimensional prob-
lems, such as 1D blood vessel flow coupled to three-dimensional (3D) fluid flow (see
Figure 3.2e). Clearly, the reverse of this mapping is ill-posed and can only be obtained
in simplified instances.
Definition 3.7 (degenerate interface). An interface Ii→j={Ti,T
j,ei,j ,Ii,j }
is called degenerate if for some (e1,ξ1),(e2,ξ2)∈{1,...,N
E,i}×τithere exists a
single pair (e′,ξ′)∈{1,...,N
E,j}×τjsuch that ζi→j(e1,ξ2)=(e′,ξ′),ζi→j(e2,ξ2)=
(e′,ξ′).
An object-set of well-defined interfaces plays a crucial role in reusability of
a single-physics code within coupled problems, as it enables the abstraction of
interpolation-specific implementations as well as cross-talk between different com-
putational domains.
3.4. Variables and function spaces. Using the basis, topology, and interface
structures outlined, it is possible to formally construct interpolated variables as well
as discrete function spaces. As with previous structures, an object-set of variables V
may be defined, enabling the construction or calculation of numerous quantities that
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C161
are either required for solving physical systems or desired as outputs. Each variable
object V∈Vis defined as V={TV,CV},whereTV={EV,B
V,N
E,N
N}∈Tis the
topology, and CV∈Rn×NNthe coefficient matrix, so that a finite-element variable
can be computed as
(3.3) ˆvh(e, ξ)=
n
&
l=1
K
&
k=1
CV(l, EV(e, k))ϕk(ξ)el,(e, ξ)∈{1,...,N
E}×τV,
where {ϕ1,...ϕ
K}∈BV,e1,...,en∈Rnare the unit base vectors for Rn.Herewe
add a hat (ˆ), distinguishing the variable from those introduced in section 2, as it is
principally a function of the element index and master element coordinate. All fields
in CHeart, including the mesh coordinates, are specified in this way, enabling the use
of any topology defined for these fields. In the case of spatial domains, this flexibility
allows for the construction of standard simplex and hexahedral domains as well as
the use of curvilinear grids. We assume each domain variable D={CD,T
D}∈V
is well constructed (i.e., continuous, nonoverlapping, locally invertible), enabling the
construction of a computational domain,
(3.4) Ωh:= +x∈Rd,,x=ˆx(e, ξ)∀(e, ξ)∈{1,...,N
E}×τD-,
where the definition of ˆx follows from (3.3) using the appropriate domain topology
TD(and its components) as well as the scaling array CD.
However, unlike standard variables, domains have additional properties enabling
them to represent different spatial reference frames, for example, Eulerian/Lagrangian
[55] or ALE [21]. This is accomplished by allowing the domain Ωhto be considered
as a function of time Ωh(t) where the spatial positions of x∈Ωhare summed with
an additional variable V={TV,CV}∈Vto give the current x(t)∈Ωh(t), i.e.,
x(t)=ˆx(e, ξ,t)=
d
#
l=1 !KD
#
k=1
CD(l, ED(e, k)"ϕD,k(ξ)+
KV
#
k=1
CV(l, EV(e, k))$
$tϕV,k(ξ))el,
where {ϕm,1,...,ϕ
m,Km}∈Bmfor m={D, V }.WenotethatID→Vis assumed to
be an identity interface to enable summing.
Vari a b l e- s p a c e pai r i n g. While (3.3) outlines the nth-dimensional vector func-
tion ˆvhas a function of local element index and local coordinates, it is desirable
in FEMs to instead define vh:Ω
h→Rnas a function of the spatial domain Ωh.
Whether this definition is possible depends on if the variable-space pairing (ˆvh,Ωh)
is well defined.
Definition 3.8. Avariable-spacepairing(ˆvh,Ωh)(defined in (3.3) and (3.4))
is well defined if the spatial domain Ωhis well constructed (i.e., continuous, non-
overlapping, locally invertible), and there exists a topology Tr∈Tand interfaces
Ir→D,I
r→V∈I,whereIr→Dis complete and Ir→Vis an identity or injective inter-
face. In this case, Tris called a root topology.
The stipulations of Definition 3.8 ensure that for each x∈Ωhthere exists a unique
corresponding pair (e, ξ)∈{1,...,N
Er}×τr. The existence of a root topology Tras
outlined implies that the composition
(3.5) vh(x)=ˆvh◦ζr→V(e, ξ),x=ˆx ◦ζr→D(e, ξ)
is well defined for all x∈Ωh. As a result, Trprovides a topology for which vhand
Ωhcan be related. The completeness requirement for Ir→Ωensures that each xhas
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C162 LEE ET AL.
acorrespondingvh;however,Ir→Vneed not be complete, allowing for conditions or
weak equations on subsets of a variable field (i.e., use of constraints along boundaries
or interior regions).
Discrete function spaces. Using the definition of discrete variables as shown
in (3.3), and the concept of variable-space pairings introduced in Definition 3.8 and
(3.5), it is natural to define the discrete function spaces introduced in (2.1) as
Vh(Ωh)=.vh∈[L2(Ωh)]n,,vh(x)=ˆvh◦ζr→V(e, ξ),x=ˆx ◦ζr→D(e, ξ)
∀(e, ξ)∈{1,...,N
Er}×τr,∀CV∈Rn×NN/.
The properties of the discrete space Vh(Ωh), such as the smoothness and order, depend
on the underlying connectivity of TVgiven by EV, the choice of basis, as well as the
spatial domain over which the space is defined.
3.5. Expressions. Often during a modeling process, it is desirable to conduct
algebraic computations with the variables defined over the finite-element meshes at
runtime. This functionality is implemented in CHeart through an object-set of plain-
text mathematical expressions provided at problem specification time. The expres-
sions can take variable pointers, time, and other expressions as inputs and produce
outputs that can be assigned to set boundary conditions, material and source fields,
and other variables as part of partitioned solver strategies and postprocessing. The
parsing of expressions is implemented using an internal bytecode-compiler.2Based
on an algebraic expression provided by the user, the internal compiler decomposes
the string into its component operators which are then stored as a sequence of simple
bytecode instructions. This conversion occurs only once, when the problem is set up,
and so in this way the expressions can be evaluated at key points in the computation
phase of execution with a minimal amount of runtime overhead. The user can specify
both scalar and vector expressions that contain standard algebraic and trigonomet-
ric functions as well as logical and conditional operators, modulo max/min and data
interpolation.
3.6. General weak form systems. Multiphysics systems are comprised of mul-
tiple weak form operators which act on different variables. These operators, outlined
for specific physics in section 2, are constructed as needed and built into an object-set
F,whereeachoperatorf∈Fis defined as in (2.1).
Remark 3.9. In the case of Lagrangian or arbitrary Lagrangian–Eulerian (ALE)
domains, functionals can be defined as
f(vh,yh;Ω
h(t)) = !Ωh(t)
a(vh)·yh+c(vh):∇yhdΩh,
enabling definition of the weak form on a specific reference frame. In the case of
conservative ALE formulations [25, 70],
f(vh,yh;Ω
h)=!Ωh(t+δt)
a0(vh)·yhdΩh−!Ωh(t)
a1(vh)·yhdΩh
+!t+δt
t!Ωh(s)
a2(vh)·yh+c(vh):∇yhdΩhds
2http://fparser.sourcefor ge.net/
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C163
is used with functionals a0,a1,anda2.
Remark 3.10. Boundary conditions may be introduced to operators f∈Fas
discussed in section 4.2.
Each operator f∈Fmay be combined with others to form multiphysics systems.
This is done by defining the summed operator f⊆F,
(3.6) f(vh,yh) := &
k
fk(vh,k,yh,k ;Ω
h,k),f
k∈F,
where fkand Ωh,k are the kth operator/domain and (vh,k ,yh,k) its respective variables
(where vh,k ∈vhand yh,k ∈yh). Importantly, the operator fand its variables,
vh,yh, do not reside on a single domain. This enables the merger of operators which
span different Ωh,k,asoftenoccursinmultiphysics systems. Each operator fdenotes
aspecificsystemforwhichwelooktofindvh∈Vhso that
(3.7) f(vh,yh)=0 ∀yh∈Yh,
where Vh,Yhdenote the complete trial/test spaces.
4. Physics coupling. In sections 2 and 3 we introduced the formulation of
single-physics operators as well as their construction through a series of object-sets.
To further enable flexible multiphysics integration, this section discusses a series of
coupling strategies developed to formally link operators (as shown in (3.7)) which
represent multiphysics weak form systems. Here we define a coupling as a functional
relationship between single-physics systems within a problem, the examples of which
can be found in Table 4.1. While this may occur in numerous instances, here we
identify three principal mechanisms:
(1) functional dependence of existing operators on evolving variables from an-
other problem,
(2) functional dependence of boundary conditions on evolving variables from an-
other problem, and
(3) new coupling operators which effectively link problems together.
These approaches are all made possible using different couplings methods.
4.1. Coupling via existing operators. As full access to all variables in V
is available to any problem, the inclusion of variables through existing operators is
straightforward. For example, the diffusion coefficient in a scalar transport problem
may be linked to any variable in V, including one which is solved for within a different
system (e.g., as is the case in strongly coupled electromechanics). This enables the
automatic coupling of systems. Further, as different solve strategies may be applied as
discussed in section 5.5, CHeart enables this coupling to be either explicit or handled
through a fixed point iteration.
4.2. Coupling via boundary conditions. Each operator (see section 3.6) may
be linked to any number of boundary conditions. Steady or time-dependent Dirichlet
or Neumann boundary conditions may be applied to different portions of the domain
boundary (typically a series of user-defined nodes or element faces).
CHeart boundary conditions may be specified using variables as well as numeric
expressions, so that the solution of one problem may be provided as the boundary
data for another problem. This is often required in partitioned ALE Navier–Stokes
where the domain motion can be computed independently from the flow while the two
problems are coupled along portions of the domain boundary. This type of coupling is
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C164 LEE ET AL.
Tab l e 4 . 1
Summary of available coupling options for different styles of multiphysics.
Physics coupling operators/options
Simple coupling Coupling via share d variables
Coupling via algebraic expressions
Coupling via b oundary conditions
Lagrange coupling Coupling one or m ore variables to a constraint
Coupling variable or variable rate, i.e., Du=uor Du=∂u/∂t
Coupling all components or norm al components, i.e., L(u)=uor L(u)=u·n
User defined constraint expressions available
Support for constraints on 1D, 2D, and 3D manifolds or volumes
Volume coupling Linear coupling through reaction terms
Support for 2D, 3D
also common in FSI, where the motion of the interface between fluid and solid domains
is set as a Dirichlet condition and the traction on fluid is applied as a Neumann-type
condition to the solid.
4.3. Coupling via expressions. While the integration of multiple physics us-
ing variables provides a powerful tool for coupling systems, this is not always sufficient
as the coupling may require algebraic or differential manipulation of a variable. This
feature is enabled in CHeart via expression strings (see section 3.4) which may be
provided by users to define the specific mathematical form of coupling.
In many cases, such as with boundary conditions, expressions and variables may
be used interchangeably. In addition, CHeart allows the casting of expressions to
variables, which enables the coupling of certain operations which would otherwise
need to be hardcoded.
4.4. Monolithic integration via coupling operators or Lagrange multi-
pliers. Multiphysics integration often necessitates the inclusion of additional terms
or constraints in the system. In some cases, as in partitioned approaches, this merely
involves adding a source or augmenting boundary conditions. In other approaches,
the coupling may be imposed via cou pl ing o perat ors ,orbytheintroductionofLa-
grange multipliers. In these cases, the additional elements are considered as separate
physical problems which augment traditional single-physics examples. The modular
implementation of these couplin g problems in CHeart promotes code reuse in various
types of multiphysics problems.
Coupling operators. In CHeart coupling operators provide a link between
variables from distinct physics problems through the addition of new terms or opera-
tors into the weak form. Considering a set of variables (u1,...,um)∈Uh
1×···×Uh
m
which have been introduced for a set of physics problems, the general form of ain
equation (2.1) for coupling operators is given by
a(u1,...,um) := $m
&
k=1
γ1kL1kuk, ...,
m
&
k=1
γmkLmk uk%,
where γik is a scalar coefficient detailing the coupling between uiand ukand Lik
the corresponding linear operator (see Table 4.1). As a result, the weak form defined
for uiis augmented with 0m
k=1 γikLik uk,whichmaybetailoredtoaddressaspecific
type of coupling. One classical example of the use of coupling operators is in Boussi-
nesq convection, where the Navier–Stokes equations are coupled to advection-diffusion
through the advective velocity field and a temperature-based forcing term.
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C165
Lagrange multipliers. Lagrange multipliers are typically used for the addition
of new constraint equations which may augment a single-physics system or link mul-
tiple systems. Specifically, an additional variable λ∈Mhis added to the set of
variables (u1,...,um) in order to weakly impose over some domain or manifold, Ωh,
that the variables satisfy
&
k
wkLk(Dkuk)−s=0,
where Lkand Dkare the kth linear algebraic and differential operators, respectively
(see Table 4.1), wkis the kth scalar weight, and sis some provided source. The
additional constraint terms can be included using the operator aof (2.1) defined as
a(u1,...,um,λ) := $w1L1(λ),...,w
mLm(λ),&
k
wkLk(Dkuk)−s%,
where (u1,...,um,λ)∈Uh
1×···×Uh
m×Mh.Thisenablesabroadrangeofcoupling
using multipliers and may also be used to construct projection operators for iterative
fixed point schemes. Examples of this approach are found in the imposition of complex
physical constraints as well as fluid-solid coupling.
5. Assembly, matrix operations, and building. Efficient and flexible ma-
trix assembly operations are a principal enabler of multiphysics integration in finite-
element modeling and a key engine driving its computational performance. The core
CHeart library provides automated functionalities and standardized interfaces to en-
able rapid and extensible developments. The process by which the topology group,
equation map, and variable are used to build and populate sparse matrices is described
below.
5.1. FEM element assembly. FEM element assembly of residuals, matri-
ces, and other objects (see section 3) is usually handled by an element loop over
amesh. Inthecode,CHeart contains a single element loop routine, using a series of
procedure pointers—Setup,LocalElementCalculate,LocalElementAssemble,and
CleanUp procedures—to control the tasks executed. These procedure pointers may
be linked (or nullified) before execution of the element loop and altered accord-
ing to the desired assembly task (i.e., residual evaluation, Jacobian construction,
norm calculations, etc.). Setup and CleanUp procedures are commonly used to ex-
tract tasks which need not be repeated during the element loop—such as dynami-
cally allocating/deallocating memory or constructing/destroying access shortcuts to
data required for FEM assembly—to optimize computational performance. The
Calculation and Set routines are established to do tasks which may vary by ele-
ment, with the Calculation step involving the computation of a physics-dependent
element-level matrix or residual, for example. The Set step is a low-level, abstracted
task used to incorporate element-level data into global matrix structures.
Beyond the standard steps followed in FEM assembly, the presence of merged
functionals and interface mappings necessitates important extensions. As the weak
form functional for a physical system may be a sum of multiple physical systems
(see (3.6)), the general assembly occurs over each individual functional fk∈f,for
which we define a symbolic variable dependence using equation maps, E(fk). Cru-
cially, this equation map contains a root topology Tr, over which element-level proce-
dures may be computed.
Since integral evaluations of weak form equations (see (2.1)) are computed over
Tr, all other topologies used as part of the problem governed by fkmust be evaluated
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C166 LEE ET AL.
on Tr. This list may be automatically compiled by recording the usage of various
topologies as problems are defined by the user. The completeness of Trensures the
looping process covers the indicated spatial domain.
In addition, the corresponding basis functions and their gradients of all involved
terms must be evaluated on Tr.IfIr→kfor Tk∈Tis an identity interface (see
Definition 3.1), no mapping is required, and pre-evaluations of the basis functions
(and their derivatives) can be used directly. However, when Ir→kis a more complex
mapping, such as an injective interface, then the basis on Tkmust be mapped onto
Tr(see (3.6)). Users can elect to either execute these computations on the fly or have
the program precompute these quantities into fast access cached lists.
Algorithm (FEM LOOP)
for all fk∈f
call SetupProcedure
for all e∈Tr⇐E(fk)
for all Ti∈{Tk}
Compute needed ϕ,∇rϕ, . . . at Qrfor all ϕ∈Bi
end for
call LocalElementCalculateProcedure
call LocalElementAssembleProcedure
end for
call CleanUpProcedure
end for
5.2. Equation mappings. The translation of weak form operators finto dis-
crete finite-element equivalents may be achieved via the use of equation maps (Defini-
tion 5.1). An equation map is an abstraction of the dependence of weak form operators
on their spaces, which completely encapsulates the row-column variable mappings
necessary to construct the local element tensors. Considering (2.2), the equation
mapping defines whether f(v′
h,y′
h;Ω
h)̸=0forsomev′
h=(0,...,vh,i ,...,0)and
y′
h=(0,...,yh,j ,...,0)withvh,i ∈Vh
iand yh,j ∈Yh
j, that is, whether there are
operators in fwhich pair the ith and jth trial and test spaces.
Definition 5.1 (equation map). Let {Vh
1,...,Vh
m}and {Yh
1,...,Yh
n}be well-
defined function spaces on Ωh.Then,iffis a functional form as shown in (2.2) on
the discrete finite-element spaces, the equation map E(f)of fis defined for each row
variable i∈{1,...,n}by Ei(f)where
Ei(f) := {j∈{1,...,m}|f(vh,yh;Ω
h)̸=0 for some (vh,yh)∈Vh
j×Yh
i},
denoting the symbolic nonzero dependence of terms in fon {Vh
1,...,Vh
m}×{Yh
1,...,Yh
n}.
Remark 5.2. In Definition 5.1, the equation map E(f) implies that the operator,
f,operatesonwell-definedspacesonΩ
h(with topology TD∈T). Hence, there is
a root topology, Tr, which links all topologies of variables in E(f). That is, Ir→Dis
complete and Ir→Kis nonnull for each Tk⇐E(f).
Encoding the operators of f,theequationmapE(f) enables the generalization
of many lower-level routines and operations which require knowledge of the weak
form structure, such as linearized system assembly, merger, etc. In addition to the
symbolic dependence outlined, the equation map E(f)offalso labels the structure
of each term as identity,trace,orfull. This relates to the pairing of vector variables,
which, due to operators, may share components in different ways (see Definition 5.3).
For example, the standard inner-product of two vector variables may be classified as
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C167
v
p
u
q
v
u
λ
v
p
u
q
λ
Fluid mechanics
Solid mechanics
Lagrange multiplier coupling Monolithic cou
p
led s
y
stem
Fig. 5.1.Illustration of three separate equation maps—incompressible Navier–Stokes with ve-
locity and pressure variables (v,p),incompressiblesolidmechanicswithdisplacementandpressure
variables (u,q),andLagrangemultipliercoupling,λ,between(v,u)—combined into a single equa-
tion map to form the monolithically coupled system used in FSI; see section 8.2.
having trace structure, while the standard inner-product of two scalar variables is a
full structure operator.
Definition 5.3 (equation map structure). Let E(f)be the equation map of the
operator fgiven in Definition 5.1.ConsideringthevariablepairingVj×Yi,forsome
j∈Ei(f),supposeeachvh∈Vh
jand yh∈Yh
itakes vh:Ω
h→Rsand yh:Ω
h→Rr.
Letting vn,k
h∈Vh
jdenote the nth basis function in the kth component and, similarly,
letting ym,p
h∈Yh
idenote the mth basis function in the pth component, the structure
of the variable pairing is defined as
(Identity) if f(vn,k
h,ym,p
h)̸=0for some n=m, k =p,
(Trace) if f(vn,k
h,ym,p
h)̸=0for some n, m, k =p,
(Full) if f(vn,k
h,ym,p
h)̸=0for some n, k , m, p,
where the least generic structure applicable is selected for the specific functional f.
While the equation mapping of an operator enables the construction of a generic
template for FEM calculations, its dependence on a single domain, Ωh, is limiting,
particularly in multiphysics contexts where multiple domains are linked together. As
aresult,themultiphysicsoperatorfis enabled through the formation of equation
map sets where the final structure is the union of each equation map automatically
evaluated from the individual maps provided, i.e., E(f) := 1kE(fk). An example of
this process is illustrated in Figure 5.1. In these cases, operations are done on each
individual equation map, E(fk), and their results are combined to emulate the final
mapping, E(f).
5.3. Matrix assembly and monolithic integration. The final weak form
system shown in (3.7) may be solved in a variety of ways. This is achieved, in part,
by allowing single-physics or other operators to be integrated into a single monolithic
problem which may be solved implicitly. Importantly, not all operators of the total
problem fneed be monolithic, thereby allowing partitioned solving of some/all physics
systems.
For problems with monolithic integration, their implicit solution requires the for-
mulation of a Jacobian matrix (or some approximation to it) which may be inverted
or iteratively solved to give the solution or a solution update (as in Newton–Raphson
methods [57, 32]). Similar to other structures, we construct an object-set Mof ma-
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
C168 LEE ET AL.
trices. Each matrix, M∈M, is assembled on an underlying equation mapping E(f)
which may be composed of a simple physical system or a more complex multiphysics
system spanning multiple domains and variables. From the equation map definition,
we seek to assemble Min compressed sparse row (CSR) format [27] by determining
the connectivity between a series of rows and columns. In this sense, the matrix,
(5.1) M={Vr,Vc,E},
is a graph with row/column vertices (Vrand Vc)andedges(E)whichprovidetheir
connectivity. In the case of structurally simple equation maps—such as that which
arises from scalar Laplace—the matrix construction process is straightforward. In
this case, if the solution variable is U={U,T}∈Vwith NNnodes/modes, then by
computing the inverse mapping of the topology array E∈T(the node-to-element
map), E−1(n)={e∈{1,...,N
E}| n=E(e, k),for some k∈{1,...,K}},n ∈
{1,...,N
N}one can effectively determine the interdependence of nodes/modes on
elements. Subsequently, the inverse map can be used to directly compute the matrix
graph M={Vr,Vc,E}.HereVr=Vcis defined so that, for n∈{1,...,N
N},
Vr(U, n)=nand
(5.2)
E(n)={m∈{1,...,N
N}| m=E(e, k)forsomek∈{1,...,K}and e∈E−1(n)}.
Remark 5.4. In the case of discontinuous Galerkin assembly, ˜
E−1is defined and
used in place of E−1in (5.2) with ˜
E−1(n)={e∈{1,...,N
E}|n=E(el,k)forsome
k∈{1,...,K},and el∈N(e)},n ∈{1,...,N
N},and N(e)denotingthesetof
neighboring element indices of eand eitself.
In a multiphysics environment, this approach must be adapted to deal with ad-
ditional (potentially multidimensional) variables, multiple equation maps (i.e., f=
{fk}), and complex interfaces. However, the construction of more complex linear
systems can be efficiently managed using both the equation mapping E(f) as well as
the underlying interface mappings, I.E(f)providesdetailsofhowthesetofi-row
variables, Y={Y1,...,Y
i}⊆V, as well as j-column variables, V={V1,...,V
j}⊆V,
is interrelated. Using this data, we may construct a variable-matrix mapping for both
rows and colums. That is, for any Y={C,T}∈Yand n∈{1,...N
N},wedefinethe
row mapping Vr(Y,n)=R, R ∈{1,...,N
Vr},whereNVris the total number of rows
(and similarly for the column mapping). We note that, in this case, Ris uniquely
paired to Yand n. This definition effectively orders how nodes/modes associated
with specific variable spaces are mapped to the linear system.
Remark 5.5. Construction of Vrand Vcmappings follows element clustering,
where all variables are ordered according to elements, or variable clustering, where
each variable is ordered sequentially, depending on the parallelization strategy.
Subsequently, assembly is managed by constructing the matrix connectivity, E,as
seen in Definition 5.6. Here, the connection between row and column variables is ex-
tracted by first operating on each equation mapping E(fk) resulting from functionals
fk∈f.ThisprovidesalinktotheroottopologyTrkover which all row and column
variables of E(fk) are related. Each element of the root topology Trkthen acts as
the common point, enabling mapping through the defined interfaces to determine the
corresponding elements in the topology used by all variables in the mapping. By col-
lating these nodes/modes, it is then possible to construct the collective connectivity,
E, as given in Definition 5.6 and illustrated in Figure 5.2.
Downloaded 06/13/16 to 138.38.74.106. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.
MULTIPHYSICS COMPUTING IN CHeart C169
Fig. 5.2.An il lustration of mon olit hic matrix assemb ly procedure. Top left: T wo topologies
(quadrilateral and triangle types) with node numbers indicated. The equation maps for the individual
systems are E(f1(U)) and E(f2(V)).Whenthetwosystemsarecoupled,thecouplingoperator
provides the additional functionals, f3,andequationmaps,E(f3),linkingthevariables. Following
the formation of the monolithic E(f),thematrixconnectivityEcan be dete rm in ed thro