Content uploaded by A. Benjamin Hocking
Author content
All content in this area was uploaded by A. Benjamin Hocking on May 27, 2024
Content may be subject to copyright.
Polygon Merge: A Geometric Algorithm Verified
Using PVS
Ben L. Di Vito and Ashlie B. Hocking
Dependable Computing, Charlottesville, VA USA
{ben.divito,ben.hocking}@dependablecomputing.com
Abstract. Geometric algorithms can present significant challenges for
formal methods. We describe the formalization and verification of an al-
gorithm posing such challenges. Given two overlapping polygons A and
B, the Polygon Merge algorithm derives a new polygon whose edges are
outermost, partial edges of A and B. The algorithm has been verified to
satisfy correctness criteria expressed as two point-set properties. We have
constructed a rigorous proof using the PVS interactive theorem prover
along with support from the NASA PVS Library. While the algorithm
itself is fairly compact, its proof was a moderately complicated undertak-
ing. Owing to the complexity of reasoning about the spatial relationships
of two different polygons, the proof required over 700 supporting lemmas.
Most of the definitions and lemmas introduced were needed to build a
body of deductive artifacts sufficiently robust for this problem domain.
Many of these artifacts will be generally useful for working with poly-
gons, line segments and 2D vectors. Accordingly, we are distilling these
products into a reusable PVS library.
Keywords: Deductive verification ·Interactive theorem proving ·Geo-
metric algorithms ·Polygon membership
[Note to reviewers: At submission time, work was ongoing to complete proofs for
several remaining lemmas. We expect this work to be done by early January.]
1 Introduction
Geometry is a problem domain where human abilities in spatial reasoning and
visualization are strong assets. Conducting formal proofs of geometric problems
using interactive theorem provers is challenging simply because they cannot (yet)
match human reasoning ability. Considerable detail must be provided during
both formalization and proof to succeed in this domain. Formidable headwinds
are to be expected.
Nevertheless, for problems of sufficient complexity and applications of some
consequence, the return on investment of formal proofs is worthwhile. We have
carried out a substantial effort to formalize and verify the Polygon Merge al-
gorithm using PVS [10]. This effort led to the modeling of several geometric
concepts found useful in reasoning about polygons. Moreover, the verification
2 B Di Vito, A B Hocking
effort uncovered a latent flaw in the first version of the algorithm. This outcome
confirms once again that subtle corner cases can confound human intuition and
escape detection. Mechanized proof is sometimes needed to expose latent flaws.
A large body of lemmas has been proved and applied to the verification task.
Many of these lemmas are generally useful and could be reused in other proof
activities. We plan to create from this work one or more PVS libraries and make
them available via the NASA PVS library collection [9].
2 Polygon Merge
Given two overlapping polygons A and B, the Polygon Merge algorithm derives
a new polygon whose edges are the outermost, partial edges of A and B. Fig. 1
illustrates the concept. Polygons A and B on the left of the figure are merged
into polygon C on the right.
Fig. 1. Example of polygon merge operation.
Notice that crossing edges result in new vertices in C that are not present
in A or B. The algorithm carries out a vertex injection operation on A and B,
resulting in new polygons Am and Bm. A/B and Am/Bm have the same point
sets for their edges and interiors. The difference is that the vertices of Am/Bm
are supersets of those of A/B, and edges of A/B can be split into multiple colinear
edges in Am/Bm.
Also notice from Fig. 1 that there is a region surrounded by edges of A
and B whose points do not lie within either polygon. We refer to these regions
as “holes.” Any number of holes is possible, including zero, and all holes are
included in merged polygon C. Thus, the merge operation differs from taking
the union of A and B, which would not be representable as a polygon.
As a precondition for merging, we require that polygons A and B have at
least one interior point in common. Polygons that share only vertices or (partial)
edges are disallowed. The case of A completely containing B (or vice versa) is
allowed; C would simply be the containing polygon A.
Polygon Merge: A Geometric Algorithm Verified Using PVS 3
2.1 Classes of Polygons Supported
Polygon Merge is formalized as an idealized algorithm operating on simple, 2D
polygons with vertices represented as x-y pairs of real numbers. Rendering the al-
gorithm in executable form would introduce approximations inherent in floating
point arithmetic, the consequences of which are not addressed in this paper.
Our definitions and lemmas are organized to work with three increasingly
restrictive classes of polygons. The most general class constrains vertices to be
unique (mutually distinct). Next we add the constraints that the vertices must
number at least three and edges may not cross one another (simple polygons).
Finally, we add the constraint that vertices must appear in counterclockwise
order. This most restrictive class is what the algorithm assumes.
2.2 Vertex Injection
Before the vertices of merged polygon C can be determined, we must first carry
out a vertex injection operation to account for edge crossings in polygons A
and B. Derived polygons Am and Bm represent the injected forms of A and B
(Fig. 2). Wherever an edge of A crosses an edge of B (or vice versa), a new vertex
appears in both Am and Bm. An injected vertex could occur at the site of an
existing vertex of A or B, in which case the injection has no effect.
Injected vertices split edges of A and B into two or more colinear edges. Such
edges are normally not admitted in definitions of simple polygons. This extension
to conventional simple polygons is a necessary feature of our algorithm.
A special case occurs for edges of A and B that have an overlap of nonzero
length. The injected vertices in this case occur at the maximum extent of the
overlap. An edge of A could be overlapped by several different, colinear edges of
B. In addition, a vertex of B could lie in the interior of an edge of A, which also
gives rise to an injected vertex for A.
An important consequence of vertex injection is that edges of Am and Bm
can intersect only at their vertices. We make heavy use of this fact in our verifica-
tion. A large subset of the lemmas is concerned with characterizing line segment
intersections and overlaps as well as traits of injected vertices and polygons.
Fig. 2. Vertex injection. Fig. 3. Counterclockwise edge traversal.
4 B Di Vito, A B Hocking
2.3 Edge Traversal and Selection
After constructing polygons Am and Bm, the next task is to derive the vertices
of the merged polygon C. The algorithm carries out a traversal of the outermost
edges to collect a sequence of selected vertices. The starting point of this traversal
is the top-left vertex,V0. We first determine VAand VB, the topmost of the
leftmost vertices of Am and Bm. Choosing the topmost of the leftmost of VA
and VByields V0.
Proceeding from V0, the algorithm selects one vertex at a time by examining
the outgoing edges from the most recently selected vertex. If there is only one
outgoing edge, the endpoint of that edge is selected. Otherwise, there is one edge
from each of Am and Bm to consider. Their spatial relationship to the previous
edge determines which edge is chosen. In particular, the rightmost of the edges is
selected. If the edges overlap, they must be equal because they can only intersect
at their endpoints. Fig. 3 shows the edge traversal for our running example.
3 Correctness Requirements
Correctness requirements for the Polygon Merge algorithm focus on relating a
point’s membership in polygons A and B to membership in merged polygon C.
3.1 Point Membership in Polygons
A point pis on the perimeter of polygon Gif there exists an edge eof Gwhere p
lies on e. We use a trichotomy of mutually exclusive and exhaustive conditions,
namely, that a point is either inside, outside or on the perimeter of a polygon.
There are two well-known methods for identifying when a point lies within
a polygon. Both have their basis as algorithms, i.e., they are computable. The
first method relies on counting edge crossings [11], variously called the crossing
number or ray casting algorithm. The second method is the winding number
algorithm [5], also called the angle summation algorithm.
We use the crossing-number criterion in our formalization. In the basic ver-
sion of the method, we begin with a point pand count all of the edge crossings
along the ray that begins at pand emanates horizontally to the right. If the
number of crossings is odd, pmust be inside the polygon. We also use a variant
of the method that allows the ray to point in an arbitrary direction (Fig. 4). In
both variants, there are special-case rules needed when the ray passes through
vertices or overlaps edges.
When formalizing properties in an expressive logic, it helps not to be confined
to computable criteria for point membership. Allowing more general methods,
such as those that quantify over infinite domains, can lead to simpler proofs. We
have found it useful to introduce such a method, the serpentine ray criterion.
A serpentine ray (s-ray) is a hybrid geometric object having a tail, which is
a ray pointing in an arbitrary direction, and a body, which is a finite sequence
of connected line segments, the last endpoint of which forms the origin of the
Polygon Merge: A Geometric Algorithm Verified Using PVS 5
Fig. 4. Crossing-number criterion. Fig. 5. Serpentine-ray criterion.
tail. Let Sbe an s-ray having starting point pat the head of S. If no part
of Sintersects any edge of polygon G, then point plies outside of G. Fig. 5
illustrates this criterion and Fig. 6 shows the PVS definitions. The addition of
this existential method gives us independent criteria to determine when a point
is inside, outside or on the perimeter of a polygon.
serp_ray_outside?(S: serpentine_ray, G: simple_polygon_2d): bool =
(FORALL (i: below(S‘body‘length - 1)), (p: point_2d):
point_on_segment?(p, (# p1 := S‘body‘seq(i),
p2 := S‘body‘seq(i+1) #))
IMPLIES NOT point_on_polygon_perimeter?(G)(p)) AND
(LET origin = S‘body‘seq(S‘body‘length - 1) IN
FORALL (d: nonneg_real):
NOT point_on_polygon_perimeter?(G)(origin + d * S‘tail))
point_outside_polygon?(p: point_2d, G: simple_polygon_2d): bool =
EXISTS (S: serpentine_ray):
serp_ray_outside?(S, G) AND p = S‘body‘seq(0)
Fig. 6. Formalizing the serpentine ray criterion.
3.2 Top-Level Requirements
Formalizing correctness requirements for the merge algorithm focuses on two
properties. The first requires that if a point plies in polygon Aor Bor lies on
their perimeters, then pmust also be inside the merged polygon Cor on its
perimeter. The second property is the converse of the first, with a twist: if plies
in C, then pmust be in or on Aor B, or pmust lie inside one of the holes. Fig. 7
shows these properties rendered in PVS.
6 B Di Vito, A B Hocking
point_in_A_or_B_is_in_merge: THEOREM
FORALL (A: (ccw_vertex_order?),
B: (ccw_merge_pre_condition(A)), p: point_2d):
point_in_polygon_inclusive?(p, A) OR
point_in_polygon_inclusive?(p, B)
IMPLIES point_in_polygon_inclusive?(p, merged_polygon(A, B))
point_in_merge_is_in_either_polygon_or_a_hole: THEOREM
FORALL (A: (ccw_vertex_order?),
B: (ccw_merge_pre_condition(A)), p: point_2d):
point_in_polygon_inclusive?(p, merged_polygon(A, B)) IMPLIES
point_in_polygon_inclusive?(p, A) OR
point_in_polygon_inclusive?(p, B) OR
point_in_polygon_merge_hole?(A, B)(p)
Fig. 7. Top-level correctness requirements.
4 Formalization and Problem Domain
We have formalized the merge algorithm using a set of PVS types and function
definitions. An implementation in SPARK Ada has also been produced along
with an assurance argument for its faithful rendering.
4.1 Formalization of Geometric Features
To formalize the algorithm and to express lemmas used in its verification, we
have modeled several geometric concepts using PVS functions.
Fig. 8. Positive cross products. Fig. 9. Selecting next merge vertex.
– 2D cross product. We make extensive use of 2D vectors and their opera-
tions. While both dot products and cross products are used, we prefer cross
products for many purposes due to their properties. The 2D cross product is
defined as uxvy−uyvx. For example, u×v>0 is useful for testing whether
vis within πradians of uin the counterclockwise direction (Fig. 8).
Polygon Merge: A Geometric Algorithm Verified Using PVS 7
– Vector operations as sine and cosine. We also exploit the trigonometric
relationships for normalized (unit) vectors subtended by angle a:u×v= sin a
and u·v= cos a.
– Minimum separation distance. For every polygon there exists a constant
Dthat bounds the distance between any vertex vand any point on any edge
not adjacent to v. The PVS function min_edge_sep returns the value of D
for a given polygon.
– Angular region membership. Given that two rays xand yboth originate
at point v(usually a vertex), it is helpful to determine whether a point p
is “between” the rays xand y. This condition holds when pcan be found
with a counterclockwise sweep from xto y. Fig. 9 depicts this notion in
the context of selecting the next (rightmost) vertex for the merged polygon.
Several PVS functions capture this notion, such as the one in Fig. 10.
– Vertex wedge regions. Every vertex vhas two regions (designated inward
and outward) on the inside and outside of the polygon bounded by the
adjacent edges of vand a circular arc at distance vertex_wedge_radius,
which is set to min_edge_sep/3. Points on the edges are excluded. We also
define the apex point of a wedge as the midpoint on the circular arc.
point_between_rays?(s, e: segment_2d)(p: point_2d): bool =
LET u = s‘p1, v = s‘p2, w = e‘p2,
vu=v-u,pu=p-u,wu=w-uIN
IF cross(vu, wu) < 0
THEN cross(vu, pu) > 0 OR cross(wu, pu) < 0
ELSE cross(vu, pu) > 0 AND cross(wu, pu) < 0
ENDIF
Fig. 10. Condition for a point to lie between rays aligned with line segments.
4.2 Key Algorithm Functions
Two functions in the algorithm’s formalization are of particular interest. First
is the recursive function for finding and collecting the merge vertices (Fig. 11).
Second is the function to select the next merge vertex at each step (Fig. 12).
The recursive function carries out the edge traversal from the starting vertex
(recall Fig. 3). It is unknown initially how many edges and vertices will form the
merged polygon. Termination occurs when the next vertex returns to the starting
vertex. The fact that the algorithm closes the loop is not shown until later in the
proof development. For this reason, the function uses a secondary termination
condition, namely, an upper bound on the number of possible vertices. This is
necessary to convince PVS that the recursion is well-founded.
When there is a choice for the next merge edge, the algorithm selects the one
further right with respect to the previous edge. (The first edge is treated as a
8 B Di Vito, A B Hocking
merge_helper(Am: simple_polygon_2d, Bm: simple_polygon_2d,
first_vtx, vertex: (point_AB_vtx?(Am, Bm)),
prev_vtx: point_2d,
vtx_num: upto(Am‘num_vertices + Bm‘num_vertices)):
RECURSIVE {s: (point_seq_AB_vtx?(Am, Bm)) |
s‘length <= Am‘num_vertices + Bm‘num_vertices - vtx_num} =
LET next_vtx: point_2d = next_merge_vertex(Am, Bm, vertex, prev_vtx) IN
IF next_vtx = first_vtx OR vtx_num = Am‘num_vertices + Bm‘num_vertices
THEN empty_seq
ELSE
singleton_seq(next_vtx) o
merge_helper(Am, Bm, first_vtx, next_vtx, vertex, vtx_num + 1)
ENDIF
MEASURE Am‘num_vertices + Bm‘num_vertices - vtx_num
Fig. 11. PVS function to find the merged polygon vertices.
next_merge_vertex(Am: simple_polygon_2d, Bm: simple_polygon_2d,
vertex: (point_AB_vtx?(Am, Bm)), prev_vtx: point_2d):
(point_AB_vtx?(Am, Bm)) =
LET idx_A = find_index(Am, vertex), idx_B = find_index(Bm, vertex) IN
IF idx_A >= 0 THEN
IF idx_B >= 0 THEN
LET e2: segment_2d = edges_of_polygon(Am)(idx_A) IN
LET e3: segment_2d = edges_of_polygon(Bm)(idx_B) IN
IF prev_vtx = e3‘p2 OR point_between_rays?(e3, e2)(prev_vtx)
THEN Am‘vertices(next_index(Am, idx_A))
ELSE Bm‘vertices(next_index(Bm, idx_B))
ENDIF
ELSE Am‘vertices(next_index(Am, idx_A))
ENDIF
ELSE Bm‘vertices(next_index(Bm, idx_B))
ENDIF
Fig. 12. PVS function for selecting the next merge vertex.
special case.) In Fig. 12, this is done using the function point_between_rays?
(from Fig. 10). Spatial relationships of the edges are illustrated in Fig. 9.
4.3 Implementation in SPARK Ada
The PVS algorithms for polygon merge were designed to be directly imple-
mentable. As an example of designing an algorithm for implementation, consider
two ways of determining whether two line segments intersect. The first method
is simple to understand, but cannot be implemented directly:
are_segments_intersecting_alt?(s1, s2: segment_2d): bool =
EXISTS(p1: (is_point_on_segment?(s1))):
is_point_on_segment?(s2)(p1);
Polygon Merge: A Geometric Algorithm Verified Using PVS 9
The second method is harder to understand, but can be implemented directly
(see Fig. 13):
are_segments_intersecting?(s1: segment_2d)(s2: segment_2d): bool =
(segment_intersect_kernel(s1, s2)‘1 = Intersecting or
segment_intersect_kernel(s1, s2)‘1 = Collinear_Overlapping);
We prove the latter definition is equal to the former definition so that an
implementation of polygon merge can mirror the PVS code as closely as possible.
segment_intersect_kernel(s1, s2: segment_2d):
[segment_intersection_type, point_2d] =
LET p: point_2d = s1‘p1 IN
LET r: vector_2d = vector_from_point_to_point(s1‘p1, s1‘p2) IN
LET q: point_2d = s2‘p1 IN
LET s: vector_2d = vector_from_point_to_point(s2‘p1, s2‘p2) IN
LET r_cross_s: real = cross(r, s) IN
LET q_minus_p_cross_r: real = cross((q - p), r) IN
IF ((r_cross_s = 0) AND (q_minus_p_cross_r = 0)) THEN
LET t0: real = (s2‘p1 - s1‘p1) * r IN
LET t1: real = (s2‘p2 - s1‘p1) * r IN
LET norm_sq: nnreal = r * r IN
IF ((0 <= t0 AND t0 <= norm_sq) OR (0 <= t1 AND t1 <= norm_sq))
THEN (Collinear_Overlapping, zero_point)
ELSE (Collinear_Non_Overlapping, zero_point)
ENDIF
ELSIF ((r_cross_s = 0) AND (q_minus_p_cross_r /= 0)) THEN
(Parallel, zero_point)
ELSE
LET q_minus_p_cross_s: real = cross((q - p), s) IN
LET t: real = q_minus_p_cross_s / r_cross_s IN
LET u: real = q_minus_p_cross_r / r_cross_s IN
IF ((0 <= t AND t <= 1) AND (0 <= u AND u <= 1))
THEN (Intersecting, mk_vect2(p‘x + t * r‘x, p‘y + t * r‘y))
ELSE (Non_Parallel_Not_Intersecting, zero_point)
ENDIF
ENDIF;
Fig. 13. Implementable function to determine line intersection.
In general, our specification of PVS should be implementable in any common
programming language, but our focus has been on making it easy to directly im-
plement in SPARK Ada. Prior work on implementing PVS algorithms in SPARK
Ada [4] identified 13 issues encountered in the process. Our specification avoids
these issues, with the exception of retrenchment from the ideal reals to IEEE
floating point numbers.
10 B Di Vito, A B Hocking
5 Proof Decomposition
Our verification effort introduced and proved a range of problem-specific lem-
mas as well as many generic lemmas, potentially applicable to other tasks. The
problem-specific lemmas can be grouped into these categories:
–Proving that points in or on polygons A and B are in or on C (forward point
containment).
–Proving that points in or on polygon C are in or on A or B or appear in one
of the holes (reverse point containment).
–Proving that the selected merge vertices form a valid polygon.
–Proving that the merge algorithm is commutative.
–Proving equivalence of the point membership criteria.
Recall Fig. 7, which shows the top-level lemmas for the point containment. In
the following sections we sketch the reasoning used to arrive at these results.
5.1 Forward Point Containment
Our first step is to reduce the point containment problem to an edge contain-
ment problem. In Fig. 14, the predicate polygon_contained? formalizes edge
containment as the condition that any point on an edge of A must be either
inside of C or on C’s perimeter. The theorem in the figure establishes that edge
containment implies point containment.
polygon_contained?(A, C: simple_polygon_2d): bool =
FORALL (p: point_2d):
point_on_polygon_perimeter?(A)(p)
IMPLIES point_in_polygon?(p, C) OR
point_on_polygon_perimeter?(C)(p)
contained_membership: THEOREM
FORALL (p: point_2d, A, C: simple_polygon_2d):
polygon_contained?(A, C) AND point_in_polygon?(p, A)
IMPLIES point_in_polygon?(p, C)
Fig. 14. Edge containment implies point containment.
Thus, it suffices to show that all edges of Am and Bm are inside of C or
part of C’s perimeter. Nevertheless, this is a significant proof effort. The chain
of reasoning is roughly as follows.
First show that the wedge for the topmost, leftmost vertex is outside of C.
Next show that all outward wedges are also outside of C. To support an induction
proof of this result, we introduce wedge connector segments that are likewise
outside of the polygon (Figs. 15 and 16). This construction enables an induction
proof to propagate “outside-ness” to all outward wedges. Next we show that the
Polygon Merge: A Geometric Algorithm Verified Using PVS 11
Fig. 15. Deriving endpoints for wedge
connector segments.
Fig. 16. Propagating “outside-ness” of
vertex wedge regions.
inward wedges are inside of C. This follows in a straightforward manner. Note
that these are general results and are not specific to merged polygon C.
Now we establish that no edges of Am/Bm can pass through the outward
wedges of C, as asserted in Fig. 17. Several intermediate lemmas are needed to
carry out the proof. This implies each edge of Am/Bm either overlaps one in C
or passes through an inward wedge of C, which is an interior region of C.
merge_no_edges_in_outward_wedges: LEMMA
FORALL (A, C: (ccw_vertex_order?), B: (ccw_merge_pre_condition(A))):
FORALL (Am: (ccw_vertex_order?), Bm: (ccw_merge_pre_condition(Am))):
(Am, Bm) = inject_vertices_into_polygon(A, B) AND
C = merged_polygon(A, B)
IMPLIES no_edges_in_outward_wedges?(C, Am) AND
no_edges_in_outward_wedges?(C, Bm)
Fig. 17. No edges in outward wedges.
As stated earlier, vertex injection ensures that vertex intersections are the
only kind possible for edges of Am and Bm. One additional lemma allows us to
infer that Am and Bm are contained by C (Fig. 18). Given that A and B have
the same point sets as Am and Bm, it follows that A and B are contained by C.
5.2 Reverse Point Containment
Showing point containment in the reverse direction relies on many of the lemmas
already formulated. We must prove that polygon C contains both polygons A
and B as well as whatever holes are created by overlaying A and B. In effect,
our goal is to show that polygon C is a “tight fit” and does not admit any
extraneous points. We formalize when a point lies in a hole region using the
function in Fig. 19.
12 B Di Vito, A B Hocking
contained_if_no_protruding_edges: LEMMA
FORALL (G, H: (ccw_vertex_order?)):
(EXISTS (q: point_2d):
point_on_polygon_perimeter?(H)(q) AND
(point_on_polygon_perimeter?(G)(q) OR
point_in_polygon?(q, G))) AND
no_edges_in_outward_wedges?(G, H) AND
(FORALL (e: (edge_of_polygon?(H))):
only_vertex_intersections?(G, H, e))
IMPLIES polygon_contained?(H, G)
Fig. 18. Absence of protruding edges implies containment.
point_in_polygon_merge_hole?(A, B: simple_polygon_2d)
(p: point_2d): bool =
NOT point_in_polygon_inclusive?(p, A) AND
NOT point_in_polygon_inclusive?(p, B) AND
FORALL (S: serpentine_ray):
p = S‘body‘seq(0) IMPLIES
EXISTS (q: point_2d):
point_on_serp_ray?(q, S) AND
(point_on_polygon_perimeter?(A)(q) OR
point_on_polygon_perimeter?(B)(q))
Fig. 19. Predicate to determine when a point lies in a “hole” region.
This definition exploits the serpentine ray concept by asserting that for every
point pand every s-ray Sfrom p, there will necessarily be an intersection of S
with an edge of A or B. Given this definition, a proof is achieved by working
through the cases and drawing from the large body of lemmas already in place.
5.3 Well-Formedness of Merged Polygon
The Polygon Merge algorithm generates a sequence of vertices pulled from the
vertices of Am and Bm. It is necessary to prove that this vertex sequence V and
the edges E defined by them meet the conditions we require of valid polygons:
1) length of V is at least three, 2) edges in E do not cross, 3) vertices in V
are unique (mutually distinct), and 4) vertices in V appear in counterclockwise
order. Three of these four conditions required moderate effort to establish, but
were not particularly troublesome. The same cannot be said for condition (3).
Proving the uniqueness result arguably has been the most difficult part of the
entire verification effort.
merge_seq_has_uniq_vertex_list: LEMMA
FORALL (A: (ccw_vertex_order?), B: (ccw_merge_pre_condition(A))):
LET M = merge_seq(A, B) IN uniq_vertex_list?(M‘length)(M‘seq)
In large part, this difficulty stems from the need to work with a (partial)
sequence of vertices rather than a complete polygon. Many previously proved
Polygon Merge: A Geometric Algorithm Verified Using PVS 13
lemmas about polygons cannot be applied. Instead, we need to reason about a
vertex sub-sequence S and a hypothetical sub-polygon created by S when the
sequence folds back on itself. Lemmas were created to describe the properties of
edges and paths of edges that enter and exit polygons. In addition, we need to
split out the special cases where one polygon is contained by the other.
5.4 Merge Algorithm Commutativity
Another useful result is the commutativity of the merge algorithm. Showing that
merge_seq(A, B) = merge_seq(B, A) is helpful in this domain because many
lemmas or cases within proofs have a dual nature. They are identical except for
the substitution of B for A and vice versa. Knowing the algorithm is commutative
allows many of these dual proofs to be discharged by instantiating lemmas in
two different ways.
5.5 Equivalence of Point Membership Criteria
Fig. 20 presents two lemmas that relate the three forms of point containment
criteria used in our verification effort. Proof of the first lemma has been deferred
for now given that it is a standard result from the literature on polygons.
point_in_polygon_conds_equiv: LEMMA
FORALL (p: point_2d, v: Normalized, G: simple_polygon_2d):
point_in_polygon?(p, G) IFF point_in_polygon_arb?(p, v, G)
point_out_not_in: LEMMA
FORALL (p: point_2d, G: simple_polygon_2d):
NOT point_in_polygon?(p, G) IFF
(point_on_polygon_perimeter?(G)(p) OR
point_outside_polygon?(p, G))
Fig. 20. Equivalence properties for point membership predicates.
Proving the second lemma, which relates predicates for when a point is inside
or outside, is fairly benign in one direction; the other direction is more demand-
ing. It makes use of a construction similar to what was depicted in Figs. 15
and 16. It is necessary to exhibit a serpentine ray that is everwhere outside of
the polygon. In effect, the constructed s-ray “skims” along the perimeter of the
polygon until it reaches a point where the tail can point outward with no edges
along its path.
5.6 Supporting Lemmas
Lower-level supporting lemmas that were introduced fall into several categories.
These lemmas vary in their generality and potential for reuse.
14 B Di Vito, A B Hocking
–Extensions to properties of finite sequences
–Properties of vertex injection definitions
–Facts about topmost, leftmost vertices
–Angular-region membership lemmas
–Facts about vertex wedge regions
–Supporting lemmas for polygon well-formedness
5.7 Problems Discovered
We close this discussion by noting that two changes to the algorithm were
prompted by discoveries during proof development. First was a flaw in an earlier
version of the algorithm that did not account for certain shared vertices. Fig. 21
shows two polygons A and B sharing vertex v. This vertex is not the result of
vertex injection from an edge crossing. Instead, both A and B have a vertex at
the same point.
Fig. 21. Counterexample found in
original algorithm.
Fig. 22. Overlapping edge should not
be selected.
The previous algorithm used a function segment_enters_polygon? to de-
cide which edge to choose instead of the current approach. The upshot is that it
allowed the anomalous behavior depicted in the figure, where erroneous and non-
terminating execution was possible. Moreover, the faulty behavior would occur
only for merge_seq(B, A); correct behavior would result from merge_seq(A, B).
Attempts to formulate and prove a particular lemma led to this discovery. We
consider it unlikely that this flaw would have been uncovered via testing. In
response, we modified the algorithm so the rightmost edge is identified and se-
lected.
The second change was not due to a flaw in the algorithm. Instead, it was
a consequence of the definition of point_between_rays?, which does not hold
when the point being tested lies on one of the two edges. This could lead to
asymmetric behavior and an inability to discern that an edge is “doubling back”
on the previously selected edge and therefore should not be selected (Fig. 22).
Polygon Merge: A Geometric Algorithm Verified Using PVS 15
What is interesting is that edge e1 in the figure would never be selected by
the algorithm because it would be in the interior of the merged polygon. The
problematic case would never arise in practice. Proving this conjecture, however,
was judged to be rather difficult. As an alternative, we introduced an explicit
test for this case in the next_merge_vertex function. In Fig. 12, the disjunct
prev_vtx = e3‘p2 was added to remedy the doubling-back problem.
6 Related Work
Work on the PolyCARP family of algorithms [8] from NASA Langley was also
focused on PVS. PolyCARP aims to compute containment, collision, resolution,
and recovery information for polygons. It is intended primarily for air traffic
management applications. Key differences with our work include the use of the
winding number algorithm for point membership and a minimum angle between
polygon edges.
Other research efforts to prove geometric algorithms focus on different formal
methods tools. A method to enhance the automatability of proofs of geometric
algorithms targets TLA+2 [6]. A different approach was used to apply Coq to
the verification of a triangulation algorithm [1].
Algorithms intended for graphics applications that concern various polygon
combinations have been developed. Computing the union, intersection and differ-
ence of two polygons is the focus of one such algorithm [7]. A second algorithm
was designed to merge a set of polygons [12]. In addition, online forums for
software developers often host discussions about similar algorithms.
Convex hull algorithms, e.g., the Graham scan [3], are related algorithms for
computing vertices with containment properties. The computational geometry
section from the book on algorithms by Cormen, et al, [2] covers convex hull and
similar algorithms.
7 Conclusion
We have described an effort to formalize the Polygon Merge algorithm and var-
ious supporting concepts, then verify that the algorithm meets two correctness
requirements. All work has been conducted using the PVS specification lan-
guage and interactive theorem prover. More than 700 lemmas were formulated
and proved; more than 500 TCCs were generated and proved. The scope and
complexity of the proof effort required to conduct our verification was larger
than expected. Despite this surprising outcome, a side benefit of our work was
the development of many reusable definitions and lemmas.
Notably, a latent flaw in the first version of the algorithm was discovered
during the proof effort. An attempt to formulate and prove one of the conjectured
properties exposed anomalous behavior. This outcome confirms, once again, the
power of formal methods to illuminate dark corners of the problem space.
Concerning verification strategy, one lesson that can be drawn from our expe-
rience is to emphasize formulating lemmas on weaker structures where possible so
16 B Di Vito, A B Hocking
that well-formedness requirements for stronger structures might be more easily
established. For example, had we expressed some lemmas in terms of unstruc-
tured point sequences rather than polygons, we could have simplified the proof
of the vertex uniqueness condition for the merged polygon.
Another lesson is not to underestimate the combinatorial impact of two poly-
gons that can be overlaid and oriented in arbitrary ways. Complications from
features such as overlapping edges are considerable. A careful strategy that can
mitigate these impacts is worth pursuing, even if it requires greater effort during
early project stages.
References
1. Bertot, Y.: Formal verification of a geometry algorithm: A quest for abstract views
and symmetry in Coq proofs (2018)
2. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms
(3rd ed.). MIT Press and McGraw-Hill (2009)
3. Graham, R.L.: An efficient algorithm for determining the convex hull of a finite
planar set. Information Processing Letters 1(4), 132–133 (1972)
4. Hocking, A.B., Rowanhill, J.C., Di Vito, B.L.: An analysis of implementing PVS
in SPARK Ada. In: 2020 IEEE/AIAA 39th Digital Avionics Systems Conference
(DASC). IEEE (2020)
5. Hormann, K., Agathos, A.: The point in polygon problem for arbitrary polygons.
Computational Geometry 20(3), 131–144 (2001)
6. Kong, H., Zhang, H., Song, X., Gu, M., Sun, J.: Proving computational geometry
algorithms in tla+2. In: 5th IEEE Intl. Conf. on Theoretical Aspects of Soft. Engin.
(TASE 2011). IEEE (2011)
7. Margalit, A., Knott, G.D.: An algorithm for computing the union, intersection or
difference of two polygons. Computers & Graphics 13(2), 167–183 (1989)
8. NASA: PolyCARP, https://github.com/nasa/PolyCARP
9. NASA Langley Research Center: PVS library collection, theories and proofs avail-
able at http://shemesh.larc.nasa.gov/fm/ftp/larc/PVS-library/
10. Owre, S., Rushby, J., Shankar, N.: PVS: A prototype verification system. In:
11th Intl. Conf. on Automated Deduction (CADE). LNAI, vol. 607, pp. 748–752.
Saratoga, NY (Jun 1992)
11. Shimrat, M.: Algorithm 112: Position of point relative to polygon. Comm. ACM
5(8) (1962)
12. Zalik, B.: Merging a set of polygons. Computers & Graphics 25(1), 77–88 (2001)
