Parallel numerical solution of the time‐harmonic Maxwell equations in mixed form

Abstract

We develop a fully scalable parallel implementation of an iterative solver for the time-harmonic Maxwell equations with vanishing wave numbers. We use a mixed finite element discretization on tetrahedral meshes, based on the lowest order Nédélec finite element pair of the first kind. We apply the block diagonal preconditioning approach of Greif and Schötzau (Numer. Linear Algebra Appl. 2007; 14(4):281–297), and use the nodal auxiliary space preconditioning technique of Hiptmair and Xu (SIAM J. Numer. Anal. 2007; 45(6):2483–2509) as the inner iteration for the shifted curl–curl operator. Algebraic multigrid is employed to solve the resulting sequence of discrete elliptic problems. We demonstrate the performance of our parallel solver on problems with constant and variable coefficients. Our numerical results indicate good scalability with the mesh size on uniform, unstructured, and locally refined meshes. Copyright © 2011 John Wiley & Sons, Ltd.

Full text

Page 1

NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS
Numer. Linear Algebra Appl. (2011)
Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/nla.782
Parallel numerical solution of the time-harmonic Maxwell
equations in mixed form
Dan Li1, Chen Greif1,∗,†and Dominik Schötzau2
1Department of Computer Science, The University of British Columbia, Vancouver, BC, Canada V6T 1Z4
2Department of Mathematics, The University of British Columbia, Vancouver, BC, Canada V6T 1Z2
SUMMARY
We develop a fully scalable parallel implementation of an iterative solver for the time-harmonic Maxwell
equations with vanishing wave numbers. We use a mixed finite element discretization on tetrahedral
meshes, based on the lowest order Nédélec finite element pair of the first kind. We apply the block diagonal
preconditioning approach of Greif and Schötzau (Numer. Linear Algebra Appl. 2007; 14(4):281–297),
and use the nodal auxiliary space preconditioning technique of Hiptmair and Xu (SIAM J. Numer. Anal.
2007; 45(6):2483–2509) as the inner iteration for the shifted curl–curl operator. Algebraic multigrid is
employed to solve the resulting sequence of discrete elliptic problems. We demonstrate the performance
of our parallel solver on problems with constant and variable coefficients. Our numerical results indicate
good scalability with the mesh size on uniform, unstructured, and locally refined meshes. Copyright ?
2011 John Wiley & Sons, Ltd.
Received 20 June 2010; Revised 3 February 2011; Accepted 21 February 2011
KEY WORDS: parallel iterative solvers; saddle-point linear systems; preconditioners; time-harmonic
Maxwell equations
1. INTRODUCTION
Consider the time-harmonic Maxwell equations in mixed form in a lossless medium with perfectly
conducting boundaries: find the vector field u and the scalar multiplier p such that
∇×?−1
r∇×u−k2?ru+?r∇p = f
in ?,
(1a)
∇·(?ru)=0
n?×u =0
in ?,
(1b)
on ?,
(1c)
p =0 on ?.
(1d)
Here ?∈R3is a polyhedral domain, which we assume is simply connected with a connected
boundary ?=??, f is a generic source and n?denotes the outward unit normal on ?. The
electromagneticparameters ?rand ?rdenote the relative permeability and permittivity,respectively,
which are scalar functions of position, uniformly bounded from above and below:
0<?min??r??max<∞
and0<?min??r??max<∞.
∗Correspondence to: Chen Greif, Department of Computer Science, The University of British Columbia, Vancouver,
BC, Canada V6T 1Z4.
†E-mail: greif@cs.ubc.ca
Copyright ? 2011 John Wiley & Sons, Ltd.

Page 2

D. LI, C. GREIF AND D. SCHÖTZAU
Thewavenumberk isgivenbyk2=?2?0?0,where?0=4?×10−7(H/m)and?0=
are the permeability and permittivity in vacuum, respectively, and ??=0 is the angular frequency.
We assume throughout that k2?ris not a Maxwell eigenvalue.
The mixed formulation is a natural and well-established way of dealing with the high nullity
of the curl–curl operator. It yields a stable and well-posed problem for vanishing wave numbers
[1–5]. In this paper we thus focus on the case
1
36?×10−9(F/m)
k?1
including k=0. The latter case is of much interest in many applications, such as magnetostatics [6].
The mixed formulation readily allows for non-divergence-free data. Large values of k are of
extreme importance in wave propagation and other applications, but we have not yet studied our
proposed technique for such ranges.
Finite element discretization using Nédélec elements of the first kind [7] for the approximation
of u and standard nodal elements for p yields a saddle-point linear system of the form
?A−k2M
B
0
?
The saddle-point matrix K is of size (m+n)×(m+n) and is symmetric indefinite. The matrix
A∈Rn×ncorresponds to the ?−1
divergence operator with full row rank; M∈Rn×nis the ?r-weighted vector mass matrix; f ∈Rn
is now the load vector associated with the right-hand side in (1a), and the vectors u∈Rnand
p∈Rmrepresent the finite element coefficients. Note that A is symmetric positive-semidefinite
with nullity m.
The saddle-point matrix problem (2) inherits the properties of the continuous formulation: it
is stable in the limit case k=0 and directly deals with the discrete gradients, without a need for
further postprocessing.
In [8], Schur complement-free block diagonal preconditioners were designed for iteratively
solving system (2) with constant coefficients. These preconditioners are motivated by spectral
equivalence properties. Each iteration of the scheme requires inverting a scalar Laplacian and
solving a linear system with A+?M, where ? is a given positive parameter. There are several effi-
cient solution methods for doing so. When a hierarchy of structured meshes is available, geometric
multigrid can be applied [9]; for unstructured meshes, algebraic multigrid (AMG) approaches have
been explored in [10–12], using the smoothers introduced in [9]. See also [13,14] for an analysis
of multigrid methods and overlapping Schwarz preconditioners for A−k2M. Recently, a highly
efficient nodal auxiliary space preconditioner has been proposed in [15]; it reduces solving for
A+?M into essentially two scalar elliptic problems on the nodal finite element space. In [16],
a massive parallel implementation of the nodal auxiliary space preconditioners was developed,
which can also deal with the limit case ?=0, using a gradient projection approach.
In this paper we extend our work in [8] and develop a fully scalable parallel implementation
for efficiently solving (2) in complex domains in three dimensions. The outer iterations are based
on the approach in [8], extended to the variable coefficient case, and the inner iterations are
solved using the method of Hiptmair and Xu [15]. We use algebraic multigrid solvers for each
elliptic problem, and accomplish almost linear complexity in the number of degrees of freedom.
For our implementation, we use state-of-the-art software packages (PETSc [17], Hypre [18], and
METIS [19]) to optimize the performance of our solvers. We have also developed our own mesher
for structured meshes, and we use TetGen [20] for unstructured and locally refined meshes. We
present an extensive set of numerical experiments, solving problems with several millions degrees
of freedom. Our numerical results scale well with the mesh size on uniform, unstructured, and
locally refined meshes.
The remainder of the paper is structured as follows. In Section 2 we analyze the properties
of the discrete operators. The preconditioning approach is presented in Section 3. In Section 4
BT
?
? ??
K
?u
p
?
=
?f
0
?
.
(2)
r-weighted discrete curl–curl operator; B∈Rm×nis the ?r-weighted
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 3

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
we provide numerical examples to demonstrate the scalability and performance of the proposed
solvers. Finally, we draw some conclusions in Section 5.
2. FINITE ELEMENT DISCRETIZATION
To discretize problem (1), we partition the domain ? into shape-regular tetrahedra of a sufficiently
small mesh size h. The electric field is approximated with Nédélec elements of the first family and
the multiplier is approximated with nodal elements of order ? [4,7]. We denote the two resulting
finite element spaces by Vhand Qh, respectively. On Vhwe enforce the homogeneous boundary
condition (1c), whereas on Qhwe impose (1d). Figure 1 shows the degrees of freedom on the
lowest order Nédélec elements in 2D and 3D.
Let ??j?n
Vh=span??j?n
Then, the weak formulation of (1) yields a linear system of the form (2), see [4, 8], where the
entries of the matrices and the load vector are given by
j=1and ??i?m
i=1be finite element bases for the spaces Vhand Qhrespectively:
Qh=span??i?m
j=1,
i=1.
(3)
Ai,j=
?
?
?
?
?
?−1
r(∇×?j)·(∇×?i)dx,
1?i, j?n,
Mi,j=
?
?r?j·?idx,
1?i, j?n,
Bi,j=
?
?r?j·∇?idx,
1?i?m,
1?j?n,
fi=
?
f ·?idx,
1?i?n.
Let us introduce a few additional matrices that play an important role in this formulation. First,
note that ∇Qh⊂Vh, and define the matrix C∈Rn×mby
n ?
For a function qh∈Qhgiven by qh=?m
∇qh=
i=1
∇?j=
i=1
Ci,j?i,
j =1,...,m.
(4)
j=1qj?j, we then have
n ?
m ?
j=1
Ci,jqj?i,
(a)
(b)
Figure 1. A graphical illustration of the degrees of freedom for the lowest order Nédélec
element in 2D and 3D. Degrees of freedom are the average value of tangential component
of the vector field on each edge. (a) 2D and (b) 3D.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 4

D. LI, C. GREIF AND D. SCHÖTZAU
so Cq is the coefficient vector of ∇qhin the basis ??i?n
of C are
i=1. In the lowest order case, the entries
Ci,j=
⎧
⎪⎩
⎪⎨
1 if node j is the head of edge i,
−1
0
if node j is the tail of edge i,
otherwise.
Define the ?r-weighted scalar Laplacian on Qhas L=(Li,j)m
i,j=1∈Rm×mwith
Li,j=
?
?
?r∇?j·∇?idx.
(5)
Finally, we set Q=(Qi,j)m
i,j=1∈Rm×mas the ?r-weighted scalar mass matrix on Qh, that is
?
Let usstateafew stabilityresultsthatextendtheanalysisin[8]fromconstantmaterialcoefficients
to the variable coefficient case.
Denote by ?·,·? the standard Euclidean inner product in Rnor Rm, and by null(·) the null space
of a matrix. For a given positive-(semi)definite matrix W and a vector x, we define the (semi)norm
?
Proposition 2.1
The following stability properties of the matrices A and B hold:
Qi,j=
?
?r?j·?idx.
|x|W=?Wx,x?.
(i) Continuity of A:
|?Au,v?|?|u|A|v|A,
u,v∈Rn.
(ii) Continuity of B:
|?Bv,q?|?|v|M|q|L,v∈Rn, q∈Rm.
(iii) The matrix A is positive definite on null(B) and
?Au,u???|u|2
M,
u∈null(B)
with a stability constant ? which is independent of the mesh size.
(iv) The matrix B satisfies the discrete inf–sup condition
inf
0?=q∈Rm
sup
0?=v∈null(A)
?Bv,q?
|v|M|q|L?1.
Proof
The first two properties follow directly from the Cauchy–Schwarz inequality.
To show (iii), we first recall the discrete Poincaré–Friedrichs inequality from [3, Theorem 4.7].
Let u∈null(B) and let uhbe the associated finite element function. Then, we have
?
where ?>0 is independent of the mesh size.
Consequently, we bound ?Au,u? as follows:
?
where ?=?/(?max?max).
?
|∇×uh|2dx??
?
?
|uh|2dx,
?Au,u?=
?
?−1
r|∇×uh|2dx?
?
?max?max|u|2
M=?|u|2
M,
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 5

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
To prove (iv), let 0?=qh∈Qhand v be the coefficient vector of vh=∇qhin the basis ??i?n
Then it follows that v∈null(A) and
?Bv,q?
|v|M|q|L
i=1.
sup
0?=v∈null(A)
=
sup
0?=v∈null(A)
?
??rvh·∇qhdx
??rvh·vhdx?1
??
2|q|L
?|q|2
|q|2
L
L
=1,
which shows (iv).
?
The properties stated in Proposition 2.1 and the theory of mixed finite element methods [21,
Chapter 2] ensure that the saddle-point system (2) is invertable (provided that the mesh size is
sufficiently small).
3. THE SOLVER
To iteratively solve the saddle-point system (2) we use MINRES [22] as an outer solver. This is
discussed in Section 3.1. To solve each outer iteration, we apply an inner solver based on [15]
and presented in Section 3.2. In Section 3.3, we outline the complete solution procedure.
3.1. The outer solver
Following the analysis for constant coefficients in [8], we propose the following block diagonal
preconditioner to iteratively solve (2):
?PM
0
PM,L=
0
L
?
,
(6)
where
PM= A+?M
(7)
and
?=1−k2>0 (8)
since k?1.
By proceeding as in [8, Theorem 5.2], we immediately have the following result; see [23] for
further details.
Theorem 3.1
Suppose k?1, the preconditioned matrix P−1
−1/(1−k2), each with algebraic multiplicity m. The remaining eigenvalues satisfy the bound
?−k2
?+1−k2<?<1,
where ? is the constant in Proposition 2.1 (iii).
M,LK has two eigenvalues ?+=1 and ?−=
(9)
3.2. The inner solver
The overall computational cost of using PM,L depends on the ability to efficiently solve linear
systems whose associated matrices are PMin (7) and L in (5).
The linear system L arises from a standard scalar elliptic problem, for which many effi-
cient solution methods exist. On the other hand, efficiently inverting PM is the computational
bottleneck in the inner iteration. Recently, Hiptmair and Xu proposed effective auxiliary space
preconditioners for linear systems arising from conforming finite element discretizations of
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 6

D. LI, C. GREIF AND D. SCHÖTZAU
H(curl)-elliptic variational problems [15], based on fictitious spaces as developed in [24,25]. The
preconditioner is
P−1
V=diag(PM)−1+P(¯L+?¯Q)−1PT+?−1C(L−1)CT,
with ? as in (8). The matrix¯L=diag(ˆL,ˆL,ˆL) is the ?−1
ˆL denotes the ?−1
¯Q=diag(Q,Q,Q) is the ?r-weighted vector mass matrix on Q3
(4), and P is the matrix representation of the nodal interpolation operator ?curl
lowest order case, the operator ?curl
h
is based on the path integrals along edges; for a finite element
function wh∈Q3
??
where ejis the interior edge associated with the basis function ?j. We have P=[P(1), P(2), P(3)],
where P∈Rn×3mand P(k)are matrices in Rn×m. The entries of P(k)are given by
?
0 otherwise,
(10)
r-weighted vector Laplacian on Q3
h, where
r-weighted (rather than ?r-weighted) version of L in (5). Furthermore, the matrix
h, C is the null-space matrix in
:Q3
h
h→Vh. In the
hit is given by
?curl
h
wh=?
j
ej
wh·d? s
?
?j,
P(k)
i,j=
0.5dit(k)
i
if node j is the head/tail of edge i,
where t(k)
edge i.
In the constant coefficient case, it was shown in [15, Theorem 7.1] that for 0<??1, the spectral
condition number ?2(P−1
no theoretical analysis available for the variable coefficient case, the preconditioner PV was
experimentally shown to be effective in this case as well [15].
i
is the kth component of the unit tangential vector on edge i and di is the length of
VPM) is independent of the mesh size. Although there seems to be
3.3. Solution algorithm
We run preconditioned MINRES as the outer solver for the linear system (2). The preconditioner
is the block diagonal matrix PM,L, defined in (6). For each outer iteration, we need to solve a
linear system of the form
?PM
0
Lq
0
??v
?
=
?c
d
?
.
(11)
Two Krylov subspace solvers are applied as the inner iterations. The linear system associated with
the (1,1) block
PMv=c
(12)
is solved using conjugate gradient (CG) with the preconditioner PV, which is defined in (10). In
each CG iteration, we need to solve a linear system of the form
PVw=r.
(13)
Following (10), this can be done by solving the two linear systems
(¯L+?¯ Q)y =s,
(14a)
¯Lz =t,
(14b)
where s= PTr and t =CTr. We run one AMG V-cycle to compute y and z, and we set
w=diag(PM)−1r+Py+?−1Cz.
(15)
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 7

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
Algorithm 1 Solve Kx =b; see (2)
1: initialize MINRES for (2)
2: while MINRES not converged do
3:
set c, d to be the right-hand-side for the current inner iteration; see (11)
4:
initialize CG for (12)
5:
while CG not converged do
6:
run one AMG V-cycle to approximate (¯L+?¯Q)−1and update y in (14a)
7:
run one AMG V-cycle to approximate L−1and update z in (14b)
8:
update w in (13), using (15)
9:
update v in (12)
10:
end while
11:
initialize CG for (16)
12:
while CG not converged do
13:
apply AMG preconditioner to approximate L
14:
update q in (16)
15:
end while
16:
update x in (2)
17: end while
The linear system associated with the (2,2) block of (11)
Lq=d
(16)
is solved using CG with an AMG preconditioner.
Our approach is summarized in Algorithm 1. The inner iteration for (12) is initialized in line
4 and laid out in lines 5–10, where CG iterations preconditioned with PV are used. The inner
iteration for (16) is initialized in line 11 and provided in lines 12–15, where a CG scheme with an
AMG preconditioner is used. Once the two iterative solvers converge, we update the approximated
solution x for the next outer iterate in line 16.
4. NUMERICAL EXPERIMENTS
This section is devoted to assessing the numerical performance and parallel scalability of our
implementation on different test cases. We use our own mesher to generate structured meshes,
TetGen [20] for unstructured and locally refined meshes and METIS [19] to partition the elements
into non-overlapping subdomains. We use PETSc [17] as the framework of our iterative solution
code. For AMG preconditioning, we use BoomerAMG [26], which is part of the Hypre [18]
package. In all experiments, the relative residual of the outer iteration is set to 1e–6 and the relative
residual of the inner iteration is set to 1e–8, unless explicitly specified. The code is executed on a
cluster with up to 12 nodes. Each node has eight 2.6GHz Intel processors and 16GB RAM.
The following notation is used to record our results: np denotes the number of processors of the
run, its is the number of outer MINRES iterations, itsi1is the number of inner CG iterations for
solving PM, itsi2is the number of CG iterations for solving L, while tsand tadenote the average
times needed in seconds for the solve phase and the assemble phase, respectively. The parameter
tAMGis the time spent in seconds in one BoomerAMG V-cycle for solving L.
4.1. Example 1
The first example is a simple domain with a structured mesh. The domain is a cube, ?=(−1,1)3.
We test both homogeneous and inhomogeneous coefficient cases. In the homogeneous case, we set
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 8

D. LI, C. GREIF AND D. SCHÖTZAU
Figure 2. Example 1. Distribution of material coefficients.
Figure 3. Example 1. (a) Structured mesh and (b) grid C1 partitioned on three processors.
?r=?r=1. In the variable coefficient case, we assume that there are eight subdomains in the cube
as shown in Figure 2 and each subdomain has piecewise constant coefficients. The coefficients are
⎧
2a
if x>0 and y<0 and z<0,
3a
if x<0 and y>0 and z<0,
?r=?r=
⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎨
⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎩
⎛
⎝
1a
if x<0 and y<0 and z<0,
4a
if x>0 and y>0 and z<0,
5a
if x<0 and y<0 and z>0,
6a
if x>0 and y<0 and z>0,
7a
if x<0 and y>0 and z>0,
8a
otherwise,
(17)
where a is a constant. We set the right-hand side so that the solution of (1) is given by
u(x,y,z)=
⎜
u1(x,y,z)
u2(x,y,z)
u3(x,y,z)
⎞
⎠=
⎟
⎛
⎜
⎜
⎝
(1−y2)(1−z2)
(1−x2)(1−z2)
(1−x2)(1−y2)
⎞
⎟
⎟
⎠
(18)
and
p(x,y,z)=(1−x2)(1−y2)(1−z2).
(19)
In this example, the homogeneous boundary conditions in (1) are satisfied.
Uniformly refined meshes are constructed shown in Figure 3(a). The number of elements and
matrix sizes are given in Table I. Figure 3(b) shows how grid C1 is partitioned across three
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 9

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
Table I. Example 1. Number of elements (Nel) and the size of
the linear systems (n+m) for grids C1–C3.
GridNel
n+m
9393931
19034163
38958219
C1
C2
C3
7146096
14436624
29478000
Table II. Example 1. Partitioning of grid C1.
Processor Local elements Local DOFs
1
2
3
2359736
2424224
2362136
3119317
3199406
3075208
Table III. Example 1. Iteration counts and computation times for various grids, k=0.
np
Grid
its itsi1
itsi2
ts (s)
ta (s)
tAMG(s)
3
6
12
C1
C2
C3
5
5
5
34
35
34
7
9
9
1,473.58
1,634.17
1,879.06
44.83
45.26
48.93
16.03
20.55
25.39
Table IV. Example 1. Iteration counts for various values of k.
k=0
itsi1
k=1
itsi1
8
k=1
itsi1
4
np
Grid
its itsi2
its itsi2
itsitsi2
3
6
12
C1
C2
C3
5
5
5
34
35
34
7
9
9
5
5
5
34
35
34
7
9
9
5
5
5
34
35
34
7
9
9
processors. Elements with the same color are stored on the same processor. Elements with the
same color are clustered together, which means that the communication cost is minimal. Table II
shows the local numbers of elements and degrees of freedom on each processor for grid C1. The
number of degrees of freedom on each processor is roughly the same, which indicates that the
load is balanced.
In our first experiment, material coefficients are homogeneous. Scalability results are shown in
Table III and the results with different values of k are shown in Table IV. To test scalability, we
refine the mesh and increase the number of processors in a proportional manner, so that the problem
size per processor remains constant. Full scalability would then imply that the computation time
also remains constant. We observe in Table III that when the mesh is refined, the numbers of outer
and inner iterations stay constant, which demonstrates the scalability of our method. The time spent
in the assembly also scales very well. The time spent in the solve increases slightly. This is because
each BoomerAMG V-cycle seems to take more time when the mesh is refined. For different values
of k, we have observed very similar computation times as in Table III. Table IV shows that the
iteration counts stay the same for the values of k that we have selected. This demonstrates the
scalability of our solver.
Setting the outer tolerance as 1e–6, we test our solver with different inner tolerances. The results
are given in Table V. We see that when the inner tolerance is looser, fewer inner iterations are
required; however if the tolerance is too loose, more outer iterations are required. For this example,
1e–6 is the optimal inner tolerance.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 10

D. LI, C. GREIF AND D. SCHÖTZAU
Table V. Example 1. Iteration counts and computation times for grid C1
on 16 processors for inexact inner iterations, k=0.
Inner tol
itsitsi1
itsi2
ts (s)
1e–10
1e–9
1e–8
1e–7
1e–6
1e–5
1e–4
5
5
5
5
6
8
43
39
35
30
21
20
16
9
8
8
7
7
6
5
557.08
505.30
440.92
383.34
375.91
409.57
794.15 21
Table VI. Example 1. Iteration counts for various values of a, k=0.
a=1
np
Grid
its itsi1
itsi2
its
a=10
itsi1
a=20
itsi1
itsi2
its itsi2
3
6
12
C1
C2
C3
21
21
21
36
35
33
8 128
128
130
33
33
33
8 238
238
236
31
31
31
8
10
11
10
11
10
11
Figure 4. Example 2. Distribution of material coefficients.
In the remaining examples, we stick to 1e–6 and 1e–8 as outer and inner tolerances, respectively.
We select a tight inner tolerance since one of our goals is to investigate the speed of convergence
of outer iterations.
Next we test the variable coefficient case. Table VI shows the iteration counts for different
variable coefficient cases. Note that the larger a is, the more variant the coefficients are in different
regions. As expected, the eigenvalue bound depends on the coefficients. Table VI shows that as
a increases, the eigenvalue bounds in Theorem 3.1 get looser and the iteration counts strongly
increase. In the variable coefficient case, both the inner and outer iterations are not sensitive to
changes in the mesh size.
4.2. Example 2
In this example, we test the problem in a complicated domain with a quasi-uniform mesh.
The domain is a complicated 3D gear, which is bounded in (0.025,0.975)×(0.025,0.975)×
(0.025,0.15292).We test two different cases: constant and variable coefficient cases. In the constant
coefficient test, we set ?r=?r=1. In the variable coefficient case, there are two subdomains as
shown in Figure 4. We have two experiments for this case: first, we assume ?r=1 in the domain,
and for x<0.5, ?r=1 and for x?0.5, ?rvaries from 10 to 1000. In the next experiment, we assume
?r=1 in the domain, and for x<0.5, ?r=1 and for x?0.5, ?rvaries from 10 to 1000. In both
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 11

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
Figure 5. Example 2. (a) Quasi-uniform mesh on the gear and (b) grid G1 partitioned on 12 processors.
Table VII. Example 2. Number of elements (Nel) and the size of the linear
systems (n+m) for grids G1–G4.
GridNel
n+m
894615
1810413
3650047
7354886
G1
G2
G3
G4
723594
1446403
2889085
5778001
Table VIII. Example 2. Iteration counts and computation times for various grids, k=0.
np
Grid
its itsi1
itsi2
ts (s)
ta (s)
tAMG(s)
12
24
48
96
G1
G2
G3
G4
4
4
4
4
58
61
64
66
8
9
9
72.11
102.27
138.87
190.19
2.50
2.56
2.55
2.68
0.42
0.67
1.05
1.5210
Table IX. Example 2. Iteration counts for various values of k.
k=0
itsi1
k=1
itsi1
8
k=1
itsi1
4
np
Grid
itsitsi2
its itsi2
itsitsi2
12
24
48
96
G1
G2
G3
G4
4
4
4
4
58
61
64
66
8
9
9
4
4
4
4
58
61
64
66
8
9
9
4
4
4
4
58
61
64
66
8
9
9
10 1010
constant and variable coefficient cases, we set the right-hand side function so that the exact solution
is given by (18) and (19), and enforce the inhomogeneous boundary conditions in a standard way.
The domain is meshed with quasi-uniform tetrahedra as shown in Figure 5(a). The number of
elements and matrix sizes is given in Table VII. Figure 5(b) shows how to partition G1 across 12
processors.
First, we test our approach with constant coefficients. The scalability results are shown in
Table VIII. The results with different values of k are shown in Table IX. We observe that when
the mesh is refined, both the inner and outer iteration counts stay constant. Again, the time spent
in the assembly scales very well. The time spent in the solve is increasing slightly, which can be
explained by the increase in tAMG. We note that the computational times for different values of k
are similar to those reported for k=0 in Table VIII. Table IX shows that the iteration counts do
not change for different k values.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 12

D. LI, C. GREIF AND D. SCHÖTZAU
Table X. Example 2. Iteration counts for various values of ?r, ?r=1, k=0.
?r=10
np
Grid
its itsi1
itsi2
its
?r=100
itsi1
?r=1000
itsi1
itsi2
its itsi2
12
24
48
96
G1
G2
G3
G4
5
5
5
5
58
60
63
65
8
9
9
9
9
9
9
47
51
54
53
8
9
18
18
17
17
50
54
54
55
8
9
10
10
10
10 10
Table XI. Example 2. Iteration counts for various values of ?r, ?r=1, k=0.
?r=10
np
Grid
its itsi1
itsi2
its
?r=100
itsi1
?r=1000
itsi1
itsi2
itsitsi2
12
24
48
96
G1
G2
G3
G4
5
5
5
5
59
64
65
65
8
9
9
7
7
7
7
61
63
62
64
8
9
9
15
15
14
14
69
64
67
65
8
9
9
10 1010
Figure 6. Example 3. Distribution of material coefficients.
Next, we test the variable coefficient example. Tables X and XI show the results for different
variable coefficient cases. Again, we observe that when the variation of the material coefficients
increases, the iteration counts increase. We also observe that both the inner and outer iterations
are not sensitive to changes in the mesh size for the example with unstructured meshes.
4.3. Example 3
The last example is the Fichera corner problem. We are interested in testing our solver on a series of
locally refined meshes. The domain ?=(−1,1)3\[0,1)×[0,−1)×[0,1) is a cube with a missing
corner. We also test both homogeneous and inhomogeneous coefficients. In the homogeneous
coefficient case, we set ?r=?r=1. In the inhomogeneous case, we assume that there are seven
subdomains in the domain as shown in Figure 6 and each subdomain has piecewise constant
coefficients. The coefficients are the same as (17). In both tests, we set the right-hand side function
so that the exact solution is given by (18) and (19), and enforce the inhomogeneous boundary
conditions in a standard way.
The domain is discretized with locally refined meshes toward the corner. Figure 7 shows an
example of a sequence of locally refined meshes. The number of elements and matrix sizes is
given in Table XII. Figure 8 shows the partitioning of F1 on four processors.
First, we assume that the material coefficients are homogeneous. The scalability results are
shown in Table XIII. The results with different values of k are shown in Table XIV. On the locally
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 13

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
Figure 7. Example 3. A sequence of locally refined meshes.
Table XII. Example 3. Number of elements (Nel) and the size
of the linear systems (n+m) for grids F1–F4.
GridNel
n+m
957277
1917649
3832895
7689953
F1
F2
F3
F4
781614
1543937
3053426
6072325
Figure 8. Example 3. Illustration of grid F1 partitioned on four processors.
Table XIII. Example 3. Iteration counts and computation times for various grids, k=0.
np
Grid
its itsi1
itsi2
ts (s)
ta (s)
tAMG(s)
4
8
16
32
F1
F2
F3
F4
5
5
5
5
59
56
58
62
8
9
204.82
213.16
253.81
314.88
6.54
6.39
6.29
6.38
0.88
1.17
1.65
1.99
10
10
refined meshes, we also observe that when the mesh is refined, both the inner and outer solvers
are scalable. Again, the time spent in the assembly scales very well. The time spent in the solve
is increasing, which is due to the increasing cost of BoomerAMG V-cycles. Table XIV shows that
the iteration counts do not change for different wave numbers. As in the previous examples, in
our experiments the computational times are also roughly the same as in Table XIII for different
values of k.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 14

D. LI, C. GREIF AND D. SCHÖTZAU
Table XIV. Example 3. Iteration counts for various values of k.
k=0
itsi1
k=1
itsi1
8
k=1
itsi1
4
np
Grid
its itsi2
itsitsi2
itsitsi2
4
8
16
32
F1
F2
F3
F4
5
5
5
5
59
56
58
62
8
9
5
5
5
5
59
56
58
62
8
9
5
5
5
4
59
56
58
63
8
9
10
10
10
10
10
10
Table XV. Example 3. Iteration counts for various values of a, k=0.
a=1
Grid
itsitsi1
itsi2
its
a=10
itsi1
a=20
itsi1
npitsi2
its itsi2
4
8
16
32
F1
F2
F3
F4
13
13
13
13
59
55
56
60
8
9
86
81
85
84
77
69
61
60
9
9
158
153
144
139
92
82
79
82
9
9
10
10
10
11
10
10
Next, we test the variable coefficient case. Table XV shows the results for different variable
coefficient cases. Again, we observe that when the coefficients vary more, the iteration counts
increase, but the scalability with respect to the mesh size is very good.
5. CONCLUSIONS
We have developed and implemented a fully scalable parallel iterative solver for the time-harmonic
Maxwell equations with heterogeneous coefficients, on unstructured meshes. For our parallel
implementation we use our own code, combined with PETSc [17], Hypre [18], METIS [19]), and
TetGen [20].
Our mixed formulation maintains a saddle-point structure, which allows for dealing with the
discrete gradients without post-processing. From a computational point of view, the saddle-point
form provides some extra flexibility, which we exploit by using an inner–outer iterative approach.
For the outer iterations we adapt [8] to the variable coefficient case. The inner iterations are based
on the auxiliary spaces technique developed in [15].
We have shown that for moderately varying coefficients, the preconditioned matrix is well
conditioned and its eigenvalues are tightly clustered; this is key to the effectiveness of the proposed
approach. As our extensive numerical experiments indicate, the inner and outer iterations are
scalable in terms of iteration counts and computation times, and the solver is minimally sensitive
to changes in the mesh size.
More work is required for making our solver robust for highly varying or discontinuous coeffi-
cients. Currently, the iteration counts are insensitive to the mesh size, but they increase when the
coefficients vary strongly. Furthermore, our results apply to low wave numbers; dealing with high
wave numbers is a primary challenge of much interest and remains an item for future work.
REFERENCES
1. Chen Z, Du Q, Zou J. Finite element methods with matching and nonmatching meshes for Maxwell equations
with discontinuous coefficients. SIAM Journal on Numerical Analysis 2000; 37(5):1542–1570.
2. Demkowicz L, Vardapetyan L. Modeling of electromagnetic absorption/scattering problems using hp-adaptive
finite elements. Computer Methods in Applied Mechanics and Engineering 1998; 152:103–124.
3. Hiptmair R. Finite elements in computational electromagnetism. Acta Numerica 2002; 11:237–339.
4. Monk P. Finite Element Methods for Maxwell’s Equations. Oxford Science Publications: Oxford, 2003.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla

Page 15

PARALLEL SOLUTION OF THE TIME-HARMONIC MAXWELL EQUATIONS
5. Vardapetyan L, Demkowicz L. hp-adaptive finite elements in electromagnetics. Computer Methods in Applied
Mechanics and Engineering 1999; 169:331–344.
6. Jin J. The Finite Element Method in Electromagnetics. Wiley: New York, 1993.
7. Nédélec JC. Mixed finite elements in R3. Numerische Mathematik 1980; 35:315–341.
8. Greif C, Schötzau D. Preconditioners for the discretized time-harmonic Maxwell equations in mixed form.
Numerical Linear Algebra with Applications 2007; 14(4):281–297.
9. Hiptmair R. Multigrid method for Maxwell’s equations. SIAM Journal on Numerical Analysis 1998; 36(1):204–225.
10. Bochev P, Garasi C, Hu J, Robinson A, Tuminaro R. An improved algebraic multigrid method for solving
Maxwell’s equations. SIAM Journal on Scientific Computing 2003; 25:623–642.
11. Jones J, Lee B. A multigrid method for variable coefficient Maxwell’s equations. SIAM Journal on Scientific
Computing 2006; 27(5):1689–1708.
12. Reitzinger S, Schöberl J. An algebraic multigrid method for finite element discretizations with edge elements.
Numerical Linear Algebra with Applications 2002; 9(3):223–238.
13. Gopalakrishnan J, Pasciak JE. Overlapping Schwarz preconditioners for indefinite time harmonic Maxwell
equations. Mathematics of Computation 2003; 72:1–15.
14. Gopalakrishnan J, Pasciak JE, Demkowicz LF. Analysis of a multigrid algorithm for time harmonic Maxwell
equations. SIAM Journal on Numerical Analysis 2004; 42:90–108.
15. Hiptmair R, Xu J. Nodal auxiliary space preconditioning in H(curl) and H(div) spaces. SIAM Journal on
Numerical Analysis 2007; 45(6):2483–2509.
16. Kolev TV, Vassilevski PS. Parallel auxiliary space AMG for H(curl) problems. Journal of Computational
Mathematics 2009; 27(5):604–623.
17. Balay S, Gropp WD, McInnes LC, Smith BF. PETSc users manual. Technical Report ANL-95/11—Revision 2.3.2,
Argonne National Laboratory, 2006.
18. Falgout R, Cleary A, Jones J, Chow E, Henson V, Baldwin C, Brown P, Vassilevski P, Yang UM. Hypre: high
performace preconditioners. Available from: http://acts.nersc.gov/hypre/, 2010.
19. Karypis G. METIS: Family of multilevel partitioning algorithms. Available from: http://glaros.dtc.umn.edu/
gkhome/views/metis/, 2010.
20. Si H. TetGen: A quality tetrahedral mesh generator and a 3D Delaunay triangulator. Available from:
http://tetgen.berlios.de/, 2010.
21. Brezzi F, Fortin M. Mixed and Hybrid Finite Element Methods. Springer: New York, NY, U.S.A., 1991.
22. Paige C, Saunders M. Solution of sparse indefinite systems of linear equations. SIAM Journal on Numerical
Analysis 1975; 12:617–629.
23. Li D. Numerical solution of the time-harmonic Maxwell equations and incompressible magnetohydrodynamics
problems. Ph.D. Thesis, The University of British Columbia, 2010.
24. Griebel M, Oswald P. On the abstract theory of additive and multiplicative Schwarz algorithms. Numerische
Mathematik 1995; 70(2):163–180.
25. Xu J. The auxiliary space method and optimal multigrid preconditioning techniques for unstructured grids.
Computing 1996; 56(3):215–235.
26. Henson VE, Yang UM. BoomerAMG: a parallel algebraic multigrid solver and preconditioner. Applied Numerical
Mathematics 2000; 41:155–177.
Copyright ? 2011 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. (2011)
DOI: 10.1002/nla