The Dynamics of Semigroups of Rational Functions I

Abstract

This paper is concerned with a generalisation of the classical theory of the dynamics associated to the iteration of a rational mapping of the Riemann sphere, to the more general setting of the dynamics associated to an arbitrary semigroup of rational mappings. We are partly motivated by results of Gehring and Martin which show that certain parameter spaces for Kleinian groups are essentially the stable basins of infinity for certain polynomial semigroups. Here we discuss the structure of the Fatou and Julia sets and their basic properties. We investigate to what extent Sullivan's 'no wandering domains' theorem remains valid. We obtain a complete generalisation of the classical results concerning classification of basins and their associated dynamics under an algebraic hypothesis analogous to the group-theoretical notion of 'virtually abelian'. We show that, in general, polynomial semigroups can have wandering domains. We put forward some conjectures regarding what we believe might be true. We also prove a theorem about the existence of filled in Julia sets for certain polynomial semigroups with specific applications to the theory of Kleinian groups in mind.

Full text

THE DYNAMICS OF SEMIGROUPS OF
RATIONAL FUNCTIONS I
A. HINKKANEN and G. J. MARTIN
[Received 4 October 1993—Revised 10 July 1995]
ABSTRACT
This paper is concerned with a generalisation of the classical theory of the dynamics associated to
the iteration of a rational mapping of the Riemann sphere, to the more general setting of the dynamics
associated to an arbitrary semigroup of rational mappings. We are partly motivated by results of
Gehring and Martin which show that certain parameter spaces for KJeinian groups are essentially the
stable basins of infinity for certain polynomial semigroups.
Here we discuss the structure of the Fatou and Julia sets and their basic properties. We investigate
to what extent Sullivan's 'no wandering domains' theorem remains valid. We obtain a complete
generalisation of the classical results concerning classification of basins and their associated dynamics
under an algebraic hypothesis analogous to the group-theoretical notion of 'virtually abelian'. We
show that, in general, polynomial semigroups can have wandering domains. We put forward some
conjectures regarding what we believe might be true. We also prove a theorem about the existence of
filled in Julia sets for certain polynomial semigroups with specific applications to the theory of
Kleinian groups in mind.
1.
Introduction
This is the first of a series of papers where we extend the classical theory of the
dynamics associated to the iteration of a rational function of a complex variable
to the more general setting of an arbitrary semigroup of rational functions. These
semigroups of rational functions may be finitely or infinitely generated and
usually will satisfy some finiteness condition. For instance, we may assume that
there are only a finite number of elements of a given degree in the semigroup.
This is the case for finitely generated semigroups.
A principal aim is to see how far the classical theory of Fatou and Julia [6,11]
applies in this more general setting. Particularly, we consider the classification of
periodic components of the Fatou set and the associated dynamics, and we are
interested in seeing to what extent such things as Sullivan's no-wandering-
domains theorem [15] remains valid and to what extent the classification theorem
for the dynamics on the Fatou set holds. Also we are interested in finding out
what new phenomena can occur. In this paper we pose a number of questions,
often phrased as conjectures, regarding the phenomena which seem to us to be
particularly interesting.
The situation in the general setting of rational semigroups is largely compli-
cated by two facts. First, the Fatou set (where the dynamics are regarded as
stable) is only forward invariant and the Julia set (where the dynamics are
regarded as unstable) is only backward invariant. Classically both these sets are
The research of the first author has been partially supported by the Alfred P. Sloan Foundation and
by the U.S. National Science Foundation grant DMS 91-07336. The second author was partially
supported by the Foundation of Research, Science and Technology, New Zealand.
This research was completed while the first author was visiting the University of Auckland,
Auckland, New Zealand. He wishes to thank the Department of Mathematics for its hospitality.
1991
Mathematics
Subject
Classification:
30D05, 58F23.
Proc. London Math. Soc. (3) 73 (1996) 358-384.

SEMIGROUPS OF RATIONAL FUNCTIONS 359
completely invariant. Secondly, a rational semigroup need not be, and most often
will not be, abelian. Classically the semigroup is actually cyclic. The simple
algebraic structure in the cyclic case implies strong consequences concerning the
possible dynamics.
A case of particular concern is that of a polynomial semigroup. Here we would
like to describe the structure of the filled-in Julia set. This is because of
applications in the theory of discrete groups (which is where the initial motivation
for this study came from). It turns out that various one-complex-dimensional
moduli spaces for discrete groups can be described as the complements of the
filled-in Julia sets for the dynamics of certain polynomial semigroups. This is
explained in [7] and the polynomials are exhibited in [8]. A well-known instance
is the moduli space of two-generator discrete free groups generated by parabolics.
We shall describe the associated polynomial semigroup in a little detail later (see
§7),
but we point out that all hyperbolic two
:
bridge knot complements appear as
points which have an essentially finite orbit under the semigroup. It is important
from many points of view to get an accurate description of these spaces. From
Teichmiiller theory, we know that the moduli space (and hence the filled-in Julia
sets) are simply connected. But an important question is whether this region is
locally connected and whether the closure is topologically a closed disk. We hope
to approach these questions in subsequent papers using a dynamical systems
approach. We offer here a first result concerning the attracting basin at infinity of
a polynomial semigroup satisfying a certain finiteness condition; see §7.
Another interesting problem is relating the algebraic structure of the semigroup
to the dynamics. As we mentioned above in the case of iteration of a rational
function, the semigroup generated is cyclic and therefore abelian. We expect that
in the generic case a semigroup will be free on its generators and therefore have a
large Julia set, often with non-empty interior. We shall try to develop criteria to
make such statements rigorous. We shall see that even simple polynomial
semigroups generated by two quadratic mappings are sufficiently complicated to
exhibit new and interesting phenomena.
We shall use A. Beardon's book [4] as a general reference for basic facts
concerning the iteration of a rational function. We also recommend this book as a
basic introduction to this theory.
We thank the referees for their helpful comments.
2.
Definitions and basic facts
A rational semigroup G is a semigroup generated by a family of non-constant
meromorphic functions,
{/i,/
2
,
...,//i> •••}> defined on the Riemann sphere C with
the semigroup operation being functional composition. We denote this situation
by
G
=
(fl,fl, •••yfn,
•••)•
Thus each g e G is a rational function and G is closed under composition. The
rational semigroup generated by a single analytic function g is denoted (g). A
rational semigroup G is called a polynomial semigroup if each g e G is a
polynomial. We denote the nth iterate of/by/".
We shall henceforth assume that a rational semigroup G contains at least one
rational function of degree at least 2.

360
A.
HINKKANEN
AND G. J.
MARTIN
All notions of convergence will be with respect to the spherical metric on the
Riemann sphere C.
A family of meromorphic functions &
is
normal in a region Q if every sequence
of elements of
2F
contains a subsequence which converges locally uniformly in Q
to a meromorphic function (possibly the constant <«) defined on Q. The set of
normality of a rational semigroup G, sometimes called the Fatou set of G, is the
largest open subset of C on which the family of analytic functions G is normal.
We denote the set of normality of G by N(G). The Julia set of G is the
complement of N(G) in C,
= C\N(G).
It is immediate that J(G) is closed. If G =
(g),
we abuse notation and write N(g)
for N((g)) and J(g) similarly. The following lemma is clear.
LEMMA
2.1. Let G be a
rational
semigroup.
Then
for
each
g e G,
(1)
(2)
We next exhibit the invariance properties of these sets. A set A is forward
invariant
under the rational semigroup G if, for each g E G,
The set is backward
invariant
if, for each g e G,
Backward invariance implies that A ^g(A) for each g e G.
THEOREM
2.1. Let G be a
rational
semigroup.
Then N(G) is forward invariant
and J(G) is backward
invariant.
Proof Let g e G. As G
°
g
= {h ° g:
h e G}
<=:
G, G
°
g is normal on N(G).
Thus G is normal on g(N(G)), hence g(N(G))^N(G), and so N(G) is forward
invariant. The second part is a consequence of the first and the fact that rational
functions map onto the Riemann sphere.
We next give a few simple examples to show that N(G) need not be backward
invariant and that J(G) need not be forward invariant. We denote the finite
complex plane by C.
EXAMPLE
1. Let a e C,
\a\
>
1
and G
=
(z
2
,
z
2
la).
Then G is a finitely generated
polynomial semigroup. We claim that
(3) J(G) = {z: l*|*|<|n|} and N(G) = {z: \z\< I or
\z\>\a\}.
First observe that both rational functions z—>z
2
and z-*z
2
/a map the regions
{z: |z|<l} and {z: |z|>|fl|} into themselves, and therefore the semigroup they
generate contains these regions in the associated set of normality N(G). Next
observe that J(z
2
)
= {z: \z\
=
1}
and J(z
2
/a)
= {z:
\z\ =
\a\}.
As J(G) is backward
invariant, we see that {z:
\z\
= \a\i}^J(G) and then, by an elementary induction,
that J(G) contains all circles of radius
\a\'
for any t = kl~" with
1 «£
k ^
2".
Since

SEMIGROUPS
OF
RATIONAL
FUNCTIONS
361
J(G) is closed, the result claimed then follows. Alternatively, one may note that
G = {z
2
"la
q
: 0^q<2
n
and n^l} and that J{z
2n
la
q
)
=
{z:
\z\
=
\a\
q
'
{V
-
l)
),
which
also implies that J{G) = {z\
1 «s
|z| ^
|fl|}.
It is clear from the above example that J(G) is not forward invariant. The
example also shows that even in simple cases the Julia set of a semigroup may
have non-empty interior even if
7(G)T^C. However, we have the following.
THEOREM
2.2. Let G be a rationaljemigroup. If J(G) is forward
invariant
and
has non-empty
interior,
then J(G) = C.
Proof.
If J(G)
¥=
C, then N(G) is open and non-empty. Let U be a non-empty
open subset of
7(G).
Then g(U) £
C\N(G)
for each g e G. Therefore the family
G is normal in U by Montel's criterion. This is impossible.
One must expect that the situation when J(G) is completely invariant for a
rational semigroup G is very special; see Conjecture 4.1. As a motivational
example we recall the Chebyshev polynomials.
EXAMPLE
2. Let T
0
(z) = 1, 7,(z) = z and
T
n+i
(z)
= 2zT
n
(z) - T
n
^(z). From the
basic fact that 7^(cos(z)) = cos(/?z), we readily deduce that G
=
{T
n
(z): n =
0,1,2,...} is a polynomial semigroup closed under composition. Also, as
T
n
(T
m
(z))
= T
nm
(z), we see that G is commutative. It is not difficult to see that
J(G) =
[-1,1]
and that in this case J(G) is completely invariant.
EXAMPLE
3. If G
=
(z
2
, 2z
2
- 1), then J(G)
=
{z:
\z\^
1}.
To see this, note that
each of z
2
and 2z
2
-l maps {z: |z|>l} into
itself,
so that {z: |z|>l}gN(G),
while J(2z
2
- 1) =
[-1,1]
and the inverse images of this interval under the
mappings z
2
" are dense in the closed unit disk. (In fact G
n
=
(z
2
,
T
n
(z))
has this
property for all n
5*
2.)
There are some general theorems in the theory of discrete groups which show
how to analyse the combination of two or more discrete groups. These are the
Klein-Maskit combination theorems; see [12]. One can use the ideas used there
to analyse the structure of the combination of two or more dynamical systems
associated to the iteration of different rational maps in some special cases. These
often provide good examples on which to test conjectures. Thus we present the
simplest possible version here. We first establish an elementary lemma on which
the result we seek is based.
LEMMA
2.2. Let G be a rational semigroup and U a non-empty open set such
that g(U)C\U
=
0 for all but finitely many geG. Then U c N(G).
Proof.
Apart from finitely many exceptional elements, G omits U on U. As U
is open, it contains more than three points. Thus G is normal on U by Montel's
criterion.
Let G be a rational semigroup and
U
a (non-empty) open backward fundamental
set for G. That is, g~\U)n U
=
0 for all g EG\ {Identity} (thus U is also a

362
A.
HINKKANEN
AND G.
J.
MARTIN
forward fundamental set). Similarly,
let
V be
a
backward fundamental
set for
the
rational semigroup
H. By the
lemma above, this implies that U
<=
N(C)
and
V^N(H)
so
that N(G)^0
and
N(H)*0. Suppose that
CW^U and
C\l/cK
THEOREM 2.3. With
G,U
and
H,V
as above,
let F
be the semigroup generated
by
G
and H,
that is,
F
=
(G, H).
Then
F
is
a
semigroup which
is
freely generated
by G
and
H
and N(F)
=>UD
Theorem
2.3
is
proved
in
the
same
way
as
Lemma
2.2.
Actually, with
a
little
care
one can
build
up
a
picture
of J(F)
from
J(G) and J{H)
using
the proof.
To provide examples
one
might,
for
instance, take
U
and
V to be
annular
regions about attracting fixed points
of
two
rational maps
/
and
g.
Then
the
semigroup
(/, g) is
free
on the two
generators.
There
are
other possibilities
for
combination theorems where
one
is
able
to
deduce
a
little
of the
structure
of the
Julia sets.
As
a
basic observation note that
if
G
and
H
are
rational semigroups
and
there
is a
set
V
for
which
f(V)
<=
V
for all
/
e
G
U
H
and if the
complement
of V
consists
of at
least three points, then
V
^
N((G,
//)).
For
instance,
if /
and
g
share
a
common attracting cycle,
an
analysis
of the
linearizations shows that
the
semigroup that they generate,
G =
(f
g),
contains
a
neighbourhood
of
this cycle
in
its set
of
normality. Thus
a
finitely generated polynomial semigroup
has
a
neighbourhood
of
oc
in its
Fatou
set.
A subsemigroup
H of a
semigroup
G is
said
to be
of
finite index
if
there
is a
finite collection
of
elements
\g
u
g
2
,...,
g
n
}
of
G
U
{Identity} such that
(4)
G
=
gi
°HUg
2
°HU...Ug
n
°H.
If
n is
chosen
to be as
small
as
possible,
we say
that
H
has
index
n in
G.
For
instance,
the
subsemigroup
H of a
finitely generated semigroup
G
consisting
of all
words
of
length some multiple
of an
integer
n has
finite index
in G.
(As,
for
instance,
the
words
of
even length
in G.)
Thus
(f
2
,
g
2
,fg,
gf) has
index
3 in (/, g).
Similarly,
we say
that
a
subsemigroup
H of
G has
cofinite index
if
there
is a
finite
collection
of
elements
g\,g2,
--.gn
of
G
U
{Identity} such that
for
every
g e G
there is
j e
{1,
2,...,
n}
such that
(5)
gj'geH.
The coindex
of H in G is the
smallest possible number
n.
Notice that
if the
semigroup were actually
a
group,
the two
notions would coincide.
In
the
example
above,
the
subsemigroup
(f
2
,
g
2
,
fg, gf) has
coindex
2 as
well
as
index
3.
The following result
is
clear.
THEOREM
2.4.
//
H is a
finite index
or
cofinite index subsemigroup
of
G,
then
N(H)
= N(G)
and
J{H)
=
J(G).
In
the
classical case
of
iteration
of a
rational function,
we see
that
the
semigroup
(/")
always
has
finite index
and
cofinite index
in (/).
A
common theme
in
the
theory when classifying
the
periodic components
of
the
Fatou
set of
/
is
to
pass
to
some iterate of/for which
the
component
is
actually fixed.
In
the
general
case
of a
rational semigroup
the
existence
of a
finite index
or
finite coindex
subsemigroup stabilising some periodic component (that
is,
mapping
the
periodic
component into itself)
is far
from clear
and is
intimately connected with
the

SEMIGROUPS
OF
RATIONAL FUNCTIONS
363
classification
of the
dynamics
on the
Fatou
set.
This
is a
problem
we
shall only
partially address
in
this paper;
see §6
where
we
establish
the
existence
of
cofinite
index stabilisers
in
nearly abelian semigroups
of
rational maps
and
relate this
problem
to the
question
of the
existence
of
wandering domains.
3.
Further properties
of Julia sets
of
rational
semigroups
In this section
we
will discuss further elementary properties
of the
Fatou
and
Julia sets which will
be
useful.
For a
given point
z
G
C we let G(z) =
{g(z):
g
G
C} denote the forward orbit
of z
under
the
semigroup
G. We
write
PL(G) for the
prelimit
set of G and L(G) for the
limit
set of G.
Here
L{G) is
defined
as the
closure
of PL(G) and a
point
z e C is in PL(G) if
there
are a
point
w e C and an
infinite sequence
of
distinct elements
{g
n
} of G
such that g
n
(w)
—>z
as
n—>
<».
(This includes
the
possibility that
g
n
(w)
=
z for all n.)
Thus
a
fixed point
for G is
in PL(G). Also
a
point
z is in PL(G) if it is
fixed
by any
element
g e G
which
is
not periodic (periodic mappings
are
easily seen
to be
elliptic Mobius transforma-
tions).
We
recall here that,
for any
rational
1
function
/, the set N(f) n PL(f)
consists precisely
of the set of all
attracting
and
superattracting fixed points
of all
iterates
of / and all
points
of
every Siegel disk
or
Herman ring occurring
for
iterates
of /
We also define
the
backward orbit
of a
point
z e C as
(6)
O~(z) =
{w
G
C:
there
is g
G
G
such that
g(w) =
z}.
LEMMA
3.1.
Let G be a
rational
semigroup.
Then
J(G)
is
perfect.
Proof.
By
assumption,
G
contains
at
least
one
element
of
degree
2 or
more.
Thus
J(G) is
non-empty
and
contains
at
least three points. Suppose that
b
G
J(G)
is
an
isolated point. Choose
a
neighbourhood U
of
b
so
that U
\
{b}
^ N(G).
Since
g(N(G)) lies
in N(G) for all g e G,
each
g e G
omits
J(G) on V
= U\{b}.
In
particular, there
is a set
consisting
of
three points that every element
of G
omits
in
V.
Thus,
by an
extension
of
Montel's theorem
(see
Caratheodory's book [5,
p.
203]),
G is
actually normal
in U so
that
b
E
N(G), a
contradiction.
Let
x e J(G) and let
U
be any
neighbourhood
of
x. Choose
any z
G
C. If
there
isnoweK
= (/\{4 for
which
g(w)
=
z for
some
g
G
G,
then
for any u
G
O~(Z)
there
is no g
G
G
with
g(w)
= u.
For if h(u)
=
z and g(w)
=
u, then
h
°
g
G
G and
(h °g)(w)
= z,
which
is
assumed
not to be the
case. Since
G is not
normal
in U
(recall that
x G/(G)),
this
can
only
be the
case
if O~(z)
contains
at
most
two
points.
For if O~(z)
contains three
or
more points, then
all g
G
G
omit them
in V
and
G
would
be
normal
in V and
hence
in U as we saw
above.
We
therefore
define
the
exceptional
set
as
(7)
E(G) =
{ZEC:
O~(Z) contains
at
most
two
points}.
We have established
the
following lemma saying that
O~(z)
clusters
at
each point
of7(G).
LEMMA 3.2. // z is not an
element
of E(G),
then
for any x
G
J(G)
there
is a
sequence
of
distinct
points
w
n
,
none
of
them
equal
to x,
tending
to x
such that
for
certain elements
g
n
e G we
have
g
n
{w
n
)
= z.

364
A.
HINKKANEN
AND G. J.
MARTIN
Thus if z e7(G) but z is not in E(G), then the closure of O~(z) is equal to
J(G) (by the backward invariance of/(G)).
For a fixed g e G let us set
(8) O~(z) =
{w
e C: g
n
(w) = z for some n >
0}.
If z is a point of E(G), then for all g e G we see that the cardinality of O~(z) is
at most 2. Thus E(G) g E(g) = E((g)) for all geC. For any g e G, other than an
elliptic Mobius transformation of order 2, there are at most two such points z
with this property. When the degree of g is at least 2 this follows from the usual
Fatou-Julia theory [4], while if the degree of g is 1 then it can be checked
directly. It follows directly that the cardinality of E(G), which we denote by
#£(G),
is at most 2. We record this as
LEMMA
3.3. Let G be a
rational
semigroup.
Then #£(G)
It is not too difficult to use the standard theory to classify the cases when
#E(G) ^ 1. In particular, if #£(G) = 2, we may assume by an appropriate
normalisation that the two points are
{0,
°°}.
It is then easily seen that every
element of the semigroup G is of the form g(z)
=
cz", where n is a non-zero
integer and c is a non-zero complex number. Any semigroup consisting entirely of
such maps indeed has this property and so this possibility can occur. Similarly
when #£(G) = 1, we can normalise so that E(G)
=
{°°}.
Now each g e G must be
a polynomial. Moreover, any polynomial semigroup has °° e E(G). In all other
cases,
we must have E(G)
=
0. If z is not an element of E(G) and 7(G)^0,
then, as we have seen, O~(z) is infinite and clusters at J(G). We have proved the
following lemma.
LEMMA
3.4. For a
rational semigroup
G the backward orbit O~(z) is
finite
for at
most two points z e C. Each such point is an element of E(G).
The next theorem can be proved following the proof given in the classical case
of iteration of entire functions. In particular, Baker's proof found in [1] (also
reproduced in [14, pp. 177-178]) generalizes easily to the case of semigroups by
replacing, in the
proof,
the iterates of a single function by general elements of the
semigroup. For completeness, we include the
proof.
THEOREM
3.1. Let G be a rational semigroup. Then the repelling cycles of
elements of G are dense in J(G).
Proof.
The repelling cycles of elements of G are contained in the Julia sets of
the individual elements of G and hence in J(G). Conversely, pick z
0
e
J(G) and
let U be a neighbourhood of
z
0
-
We need to show that U contains a repelling fixed
point of some element of G.
Since /(G) is perfect, we may find disks
Bj = B(a
Jt
e) = {z:\z~ a,\
<e}^U\
(E(G) U {z
0
})
with disjoint closures, centred at finite points
Oj
s7(G), for
1
^;^5.
We denote
the spherical derivative of a meromorphic function by /
#
; thus f*{z) =
|/(z)|
2
),
with an obvious modification if f(z) =
°°.
Let C be the

SEMIGROUPS
OF
RATIONAL FUNCTIONS
365
positive constant associated with
the set
{Bf.
1^;^5}
by the
Ahlfors theory
of
covering surfaces
as in [14, p.
177], where only three disks
are
considered
as the
functions studied there
are
analytic (without poles) rather than meromorphic.
Thus
C is so
chosen that
if /
is
any
meromorphic function defined
in the
unit disk
with
/
#
(0) >
C, then
the
unit disk contains
a
simply connected subdomain that
is
mapped by
/
conformally onto some
B
}
.
If
1
*s;
s=
5,
then
G is not
normal
in any
neighbourhood
of a
;
.
Thus,
by
Marty's
criterion
(see, for
example,
[14, p. 75]),
there
are
some
JJEG and a
point
bjzDj
=
B{aj,\e) such that /*(&,) >3C/e. Write
E
}
=
£(£>,,
^e) g£
y
.
Then
gjU)
= fj(bj
+
C)
is
meromorphic
in 5(0, \e)
with gf (0) =/*(&,) >3C/e. Hence,
as
in [1] or [14, p. 177], we
deduce that
gj
maps some simply connected
subdomain
of
B(0,\e) conformally onto some
£,-,
where I=s/ss5. Thus
the
corresponding
ft
maps some simply connected subdomain
of
£},
and
consequently
some simply connected relatively compact subdomain
of B
if
conformally onto
some
Bj.
Repeating this argument
at
most five times,
we
find some
k
with
1
ss
k
=s
5,
and an
element
g of G
arising
as a
composition
of the
jjj,
such that
g
maps some simply connected relatively compact subdomain
of B
k
conformally
onto
B
k
. It now
follows that some branch
of
g"
1
has an
attracting fixed point,
and
hence
g has a
repelling fixed point
in B
k
and
hence
in U, as
required. This
completes
the
proof
of
Theorem 3.1.
COROLLARY
3.1.
Let G be a
rational
semigroup.
Then
J(G) = U*s
COROLLARY
3.2.
Let G be a
rational
semigroup.
Then
J(G)
is the
smallest closed
backward
invariant
set containing
at
least
three
points.
Finally
we
note that
in [9] we
discuss further properties
of the
Julia
set of a
rational semigroup.
In
particular,
we
show that
the
Julia
set of a
finitely generated
rational semigroup
is
uniformly perfect
in the
sense
of
Pommerenke
[13].
Roughly this means that there
are no
short hyperbolic geodesies
in the
hyperbolic
metric
of the
Fatou
set
(defined component
by
component),
and is
related
to
regularity properties
for the
Dirichlet problem. This result
is
false
for
infinitely
generated semigroups.
4.
Nearly Abelian semigroups
In analogy with
the
theory
of
Kleinian groups,
and in
particular
the
classifica-
tion
of the
virtually abelian subgroups,
it
seems natural
to
discuss initially
semigroups
of
rational functions with
an
underlying simple algebraic structure.
One must expect that
an
abelian rational semigroup exhibits much
the
same
dynamical features
as the
classical cyclic case. However,
it
seems
to us
that
a
slightly larger family
of
rational semigroups fulfills this criterion.
We
call these
semigroups nearly abelian.
A key
example
of
such semigroups will
be the
family
of
all
polynomials having
the
same Julia
set. We
note that
the
abelian structure
of
the semigroup generated
by the
Chebyshev polynomials largely explains
the
complete invariance
of
the Julia
set in
Example 2.
In
this section
we
shall consider
what
can be
said more generally about abelian,
or
more precisely, nearly abelian,
semigroups. Before giving
the
appropriate definitions
we
first recall
the
following
well-known result concerning commuting rational functions.

366
A.
H1NKKANEN
AND G. J.
MARTIN
LEMMA
4.1.
Let
f and g be
rational
functions of
degree
at
least 2
which commute.
ThenJ(f)=J(g).
Proof.
Since
g is
uniformly continuous
on the
sphere
in the
spherical metric,
the family
{g°f
n
:
n
s*
1}
is
normal
on N(f).
This
is the
same family
as
{/"
°
g:
n^l}, and so {/": n
s*
1}
is
normal
on the
open
set
g(N(f)). Thus
g(N(/))
is a
subset
of N(f). So all the
g" omit
J(f) on N(f). As the
degree of/is
at least
2, J{f)
contains
at
least three points
and so it
follows that
the
family
{g
n
:
n ^
1}
is
normal
on N(f).
Hence
N(f)
<=
N(g).
By
symmetry
we
obtain
N(g)
c N(f).
This gives
N(f) = N(g) and
hence
J(f)
=
J(g),
as
desired.
We
say
that
a
rational semigroup
G is
nearly abelian
if
there
is a
compact
family
of
Mobius
(or
linear fractional) transformations
<I>
=
{<£}
with
the
following
properties:
(i) (f>(N(G))
= N(G) for all
<f>
G
O, and
(ii)
for
all
fg e G
there
is a 0 e
<t>
such that
f°g
= 4>°g°f
We
may
also express this definition
by
saying that
the
family
^(G) of
Mobius
transformations
(f>
for
which
f°g
= (f>°g°f
for
some
fg
E
G, is
precompact.
Note that this means that
any
sequence
of
elements
of 3>(G)
contains
a
subsequence that converges
to a
Mobius transformation uniformly
on the
2-sphere. Thus,
if
<f>
n
is any
sequence
of
elements
of
3>(G)
and D is any
disk,
it
cannot
be the
case that
4>
n
converges
to a
constant function
on D. If, in
addition,
all
the
<f>
n
have their poles outside
a
fixed disk larger than
D,
this implies
a
uniform (upper
and)
lower bound
for
\<f>'
n
\
in D.
We believe that nearly abelian semigroups provide
the
simplest examples
of
semigroups which behave
in
very much
the
same manner
as the
classical single
generator semigroups. There
are,
however, some reasonable questions
to ask
concerning
the
definition.
For
instance,
is the
supposed compactness
of the
family
O really necessary
or
reasonable?
It may
simply
be
that
the
conformal
automorphism group
of a
Julia
set is
small
and
well behaved (unless
the
Julia
set
is
the
Mobius image
of a
circle
or a
line segment,
or the
Riemann sphere), thus
making
the
compactness hypothesis redundant. However,
we
have
not
been able
to decide this matter definitively.
To prove
the
next result,
we
shall
use an
extension
of the
proof
of the
above
Lemma 4.1
and the
presumed compactness
of the
family
O.
THEOREM 4.1. Let G be a
nearly abelian rational semigroup. Then
for
each
g
e G, of
degree
at
least
2, we have
J(G)
=J(g).
Proof
Let / be a
fixed element
of G of
degree
at
least
2, and
consider
an
arbitrary element
g of G of
degree
at
least
2. Set /
=
/(/) and N
=
C\J. We
shall
show soon that
J(g)
=
J.
Assume
for a
while that this
is
true.
Now J
=
J(f) c:
J(G).
On the
other hand,
for
each
g e G, g
omits
7 on
TV
so
that
G is
normal
on
N.
Thus
N g N(G). It now
follows that
J(G) =
J,
as
claimed.
We proceed
to
prove that
if g
E
G
then
J(g)=J. For
each
n^l,
there
is
cf)
n
E <J>(G) with
f" °g
=
<t>n
o
g°f
n
-
We
begin
by
showing that g(N(f))_^N(f).
Choose some point
x e N(f) and a
neighbourhood
U of x
such that
U
9 N(f).
Then g(U)_is
a
neighbourhood
of g(x).
Consider
a
sequence
of
iterates
f
n
>
on
g(U).
As
U^N(f),
we may
pass
to a
subsequence,
say
f
m
>,
in
such
a
manner

SEMIGROUPS
OF
RATIONAL FUNCTIONS
367
that
f
m
'-^W
uniformly
on U and
where
W is
meromorphic
on U.
Since
g is
rational,
we
have g°f
m
'^g°
x
i
f
= ip
uniformly
on U.
Passing
to a
further
subsequence without changing notation,
we may
assume that
<f>
m
—> <j>
uniformly
on
the
sphere, where
</>
is a
Mobius transformation. Now
f
m
'°g
=
4>
m
°
g
°f
m
'—>
<t>
°
iff =
X uniformly
on U.
Hence
the
family
{/"
°
g:
n
s= 1}
is
normal
on U, and
so
{/": n
5*
1}
is
normal
on g(U).
Since
N(f) is the
maximal open
set on
which
{/": n
2*
1}
is
normal,
we
have
g(U) g N(f).
Thus
g(x) e N(f) and so
g(N(f))
g
N(f).
Hence every g" omits
/(/) on N(f), and so the
iterates
of g
form
a
normal
family
on N(f).
This implies that
N(f)
<=
N(g).
By
symmetry,
we
obtain
N(g)^N(f).
Hence
N(g)
=
N(f) and so 7(g) =
7(/)=7. This completes
the
proof
of
Theorem 4.1.
We note that,
if
every element
of G has
degree
at
least
2,
then
the
condition
(f>(N(G))
=
N(G) for all
<f>
e
<$>
may be
replaced
by the
condition J(f)=J(g)
for
all
f,g e G. For if
this latter property holds, there
is
clearly
a set 7 (of
cardinality
greater than
3)
such that
7 =/(/) for all / e G.
Thus each
/ e G
omits
7 in
C
\7
and
so
C\J^N(G). Since
7
= 7(/)c7(G),
we
have J=J(g)=J{G)
for all
g
e G.
Then applying both sides
of the
equation
f°g
= <f>
o
g°f to
/, we see
that
0(7)
=7 and
hence
<f>(N(G)) =
N(G). As /,g e G are
arbitrary,
the
result holds
for
all
such
<f>.
One
can ask to
what extent
the
converse
of
Theorem
4.1
holds. Often,
the
compact family
$
associated
to a
nearly abelian semigroup will
be
finite. This
is
not always
the
case,
as can
easily
be
seen with
the
nearly abelian semigroup
G
= (z
-+az
n
:
n
=2,3,...),
\a\
= 1,
arg(a)//r irrational.
In
this instance,
the
Julia
set
is the
unit circle. Here
we ask the
question
of
whether
the
family
$ is
infinite
only
in the
case that 7(G)^C
is a
circle
or
line. More pointedly:
if 7^C is the
Julia
set of a
rational function
and
4>
is the set of all
Mobius transformations
<f>
with
the
property that
$(7) =
7,
then
is it
true that
4>
is
finite unless
7
is
a
circle
or
line
or the
image
of a
line segment under
a
Mobius transformation?
A. Beardon
has
proved
the
following result which
is
clearly
of
relevance
to the
above discussion;
see [2,
Theorem
1].
THEOREM
4.2. If f and
g
are
polynomials and ifJ(f) = J(g), then
there
is a
linear
mapping
<f>(z)
= az
+
b
such
that
f
°
g
=
(f> °
g
°
f
and
\a\
=
1.
COROLLARY
4.1. Let G be a
family
of
polynomials
of
degree
at
least
2, and
suppose that there
is a set J
such that
J(g)
=
J for all g e G.
Then
(G) is a
nearly
abelian semigroup.
Proof.
As
each
g e G is a
polynomial,
7 is
compact.
As 7(/) = J(g) for all
f,g
E
(G), the
functions
/
and
g
nearly commute. This proves Corollary
4.2.
Here
is
perhaps
the
strongest result
one
could reasonably hope
for.
CONJECTURE
4.1. Let G be a
rational semigroup
and
suppose that
for
some
g
e G we
have J(g)=J(G)
and
that
J(G) is not a
circle, line
or the
Riemann
sphere. Then
G is
nearly abelian.
The family
of all
rational functions whose Julia
set is, for
instance,
the
real line

368
A.
HINKKANEN AND
G. J.
MARTIN
is rather large
and has no
apparent algebraic structure.
One can
find large
semigroups
in
this family, again with
no
apparent algebraic structure. Thus
the
hypothesis that J{G) is not
a
circle
or
line.
Given
a
nearly abelian semigroup G we let
=
{(f>:
there are f,g
e
G such that f°g~4>°g°f}.
We call <I>(G) the set
of
commutators
of
G and for f,g
e
G we write
</>
=
[/,
g]
if
f°g =
<f)°g
o
f.
Then
^(G)
is a
collection
of
Mobius transformations with
compact closure, and is contained in the group of all conformal automorphisms of
J(G).
It is
not clear
to
what extent,
if
any, there
is
further semigroup
or
group
structure
in
3>(G). Note that [/,
g]"
1
= [g,f]
so
that
^
G
$(G) implies that
<i>~
x
e
3>(G). Also
the
identity mapping, being equal
to [/,/] for
any
/,
lies
in
3>(G).
As the rational functions that we are considering here are locally invertible
outside
a
finite subset
of
the Riemann sphere,
the
commutators can locally
be
expressed using inverses. Thus many
of
the
usual commutator identities
for
groups can be verified. We note the following examples:
1.
[/,/]
= Identity;
4.
5-
[f°g>g
o
f)
o
g
o
f=f
o
g°[g>fl
These identities
can be
verified
by
straightforward calculations.
For
instance,
[f>g°f
n
] =
[f>g]
Allows because
/°g
°/"
=
[/,g]
°
g°/"
+1
and also/°go/«
=
LEMMA
4.2. Let G be a nearly
abelian
semigroup.
Then
for
all
f
e
G and for all
17 G
4>(G) there is a Mobius transformation
y
such that
(9)
f°l
=
y°f-
Moreover there are
<f>,
if/
e
$(G) such that
y
=
if/
°
(j>.
Proof.
Given
rj
e
$(G) there are
g,h
e
G
such that
g
o
h
=
t]°h°g. Then
(10)
/
o
g
°
h=f
°
T;
°
h
°
g
and furthermore, there are
<f>,il/
e
$(G) such that
(11)
f°g °h =
if/
°g
°f°h
=
if/
°
(f)
°f
°h °
g.
As f,g,h are
all
rational functions, the desired equality
/
°
TJ
= y
°
/
follows with
y
=
ifj
o (f).
We note that,
if
r/
e
$(G) is given,
it
is not always possible to find any element
y
of
the (semi)group generated by the functions
in
$(G) such that
17
°f
=f
°
y.
To see this, set G = (f, g), where
f(z)
=
z
2
- i
g
=
(f>°f,
and $(z) = -z. Then
/
has
a
parabolic cycle
at z =
-
J. Note that
/
° </>
=/
and <£
2
(z) = z,
so
that
g is
actually
a
conjugate of
/
and
so
has
a
parabolic cycle
at
2- The semigroup
G is
nearly abelian since
(12)
/
o
r/^
o
/=/
2
^
2
°/
2
=
^^/
o
/
=
^r/

SEMIGROUPS
OF
RATIONAL FUNCTIONS
369
We have <£(G)
=
{(/>,
Identity},
which
is
a
group, while
<f>
°f
= —f ¥>f =
f°y,
no
matter
how
y
e
<£(G)
is
chosen.
THEOREM
4.3.
Let G
=
(f\,f
2
,
—
,f
n
,
• ••)
be a
nearly
abelian
semigroup
and
let
H
be the group
generated
by
<£(G), that is,
H
=
(3>(G)). Then every
element
g
e
G is
expressible
in
the form
(13)
g
= yofPiofP2o_ o/P-
where
y
E
H
and the
p
(
are
non
-negative
integers.
Proof.
This follows from inductive application
of
the
above lemma based
on
the following observation:
A
o
k
°fh °fu
=k
°fh
°r/i, °fh
= fi,° <f> °
^
°
fi
2
o
fi
A
o
fh
=
4>\
°
<t>2°
«Ai
°
^2°f
it
O
fi
2
°fu°fi
3
~
y\
° Ii\ Jii °//4
Jh>
from which
it
follows that
/',
%
°fh °fu
=
72
°A
°/«4
0
A
%
=
73
% % %
%•
The following lemma
may be of
interest
in
applications
as
it
explains
to
what
extent
it
suffices
to
consider
the
commutators
of
generators.
The
proof
is
straightforward,
in
view
of the
definitions
and
Lemma 4.2,
and is
therefore left
to
the reader.
LEMMA
4.3. Suppose that
G
=
(/i,/
2
,
...,^,, •••)
is a
rational
semigroup
and
that
3>
is
a family
of
Mb'bius transformations
such that
[fi,
f
k
]
e
4>
for
all
j,k.
Then
for
all
f,g
G
G,
we
may
write
[/,
g]
=
<f>i °
... °
<f>
n
for
some
elements <f>\,...,<f>
n
ofQ?.ln
other words,
Notice that
in
Theorem
4.3, the
group
H is a
subgroup
of
the
conformal
automorphism group
of J(G).
Finally
in
this section
we
wish
to
point
out the
following result, largely contained
in A.
Beardon's work
[2,3].
THEOREM 4.4. Let
X
be a
compact set
in
the plane which
is not a
round
circle.
Let
G
=
{g:
g is a
polynomial
and J(g)
=
X).
Suppose that
G
contains
a
polynomial
of
degree
at
least
2.
Then
G is a
nearly
abelian
polynomial semigroup.
The
set ^(G)
is a
finite collection
of
elliptic transformations
and the
group
generated
by $(G) is
finite
and
cyclic.
Proof.
The
only thing
not
clear
is
that
the
commutator semigroup generates
a
finite cyclic group.
To see
this, recall that
any
commutator
<f>
e
O(G) is
a
linear
polynomial (since
it
fixes
infinity)
and
that
the
group
of
linear transformations
of
a compact
set,
which
is
not
a
round circle,
is
finite cyclic.
(To
prove this last

370
A.
HINKKANEN
AND G. J.
MARTIN
claim, notice that no
</>
e Aut(A') can be loxodromic or parabolic, since one of its
fixed points must be at <*. The group Aut(A') consists entirely of elliptic Mobius
transformations. The smallest round disk containing X is mapped onto itself by
every member of Aut(A
r
) and so they all have a common fixed point, namely, the
centre of this circle. Thus Aut(X) is closed and therefore compact. If Aut(A')
contains an irrational rotation, then X contains a circle, and therefore is a circle
by the reflection principle. Otherwise Aut(A') is finite cyclic as it is a compact
subgroup of the circle group.)
5.
Wandering
domains
In this section we shall prove that nearly abelian semigroups do not have
wandering domains and give an example which shows that in general semigroups
of rational functions may have wandering domains. The non-existence of
wandering domains is a crucial first step (though historically it was taken last!)
towards the classification of the dynamics in the Fatou set [15]. As components of
the Fatou set are only mapped into (and not necessarily onto) one another, we
must be a little careful with the definition of a wandering domain. Let U be a
component of the Fatou set N(G) of a rational semigroup G. For geGwe define
U
g
to be that component of N(G) containing g(U). Then a component U of N(G)
is called wandering if the set {U
g
: g e G} contains infinitely many elements. In
this case of course there is a sequence of elements {g,} of G such that U
gt
^ U
g
if
/ #/. In particular #,•(£/) and gj(U) lie in different components of N(G). We say
that a component U of the Fatou set is
strictly
wandering if
U
g
= U
h
implies that
g = h whenever g,h e G have degree 2 or more. That is, the forward orbit of U
consists of distinct components. For rational maps the two definitions are the
same.
For
semigroups they
are in
general different.
It
seems
to us
that
it is
possible that strictly wandering domains exist (for infinitely generated rational
semigroups) and we shall address this matter in a subsequent paper. We feel that
it is likely that a finitely generated rational semigroup has no wandering or strictly
wandering domains.
CONJECTURE
5.1. Let G be a finitely
generated
rational
semigroup.
Then G has
no
wandering
domains.
We have only partial results towards proving this conjecture in the general
setting. For instance we have shown in forthcoming work that in the expanding
case (where the closed postcritical set does not meet the Julia set) there are no
wandering domains. Our proof is based on an idea of Fatou. Also in [9] we show
how the uniform perfectness of the Julia set implies that wandering domains of
finitely generated rational semigroups cannot cluster at superattracting fixed
points of elements of the semigroup. Here we present a complete answer in the
case of nearly abelian semigroups.
THEOREM
5.1. Let G be a nearly abelian rational semigroup. Then G has no
wandering
domains.
Proof.
Let / 6 G be a rational map of degree at least 2. Then, as G is nearly
abelian, we have J(f)=J(G). Let 3>(G) be the precompact family associated

SEMIGROUPS OF RATIONAL FUNCTIONS 371
with the commutativity properties of G. Now, by Sullivan's no-wandering-
domains theorem [15], N(f) has only a finite number of components that are
periodic under / We replace / by a suitable iterate of / to assume that if U is a
periodic component of /then U is fixed, that
is,
f(U) =
U.
Let
°U
be the collection
of all fixed components of/and d the set of all components of N(G) of the form
<f>(U),
where
<f>
s 4>(G) and U e
°U.
It is easy to verify from the precompactness
of the family 3>(G), that si consists of a finite number of components of N(G).
(For instance, we may normalise so that °° e N(G) and then observe that there
are only finitely many components whose area is larger than any given constant.
Since O(G) is precompact, there is a uniform bound on the amount any element
of O(G) can decrease the area of any U e
°ti.)
We now observe that if g e G and
U
G
% then g(U) e si. To see this, simply observe that for every integer m, we
have
(14) g(U) = g(f
m
(U)) = <f>M
m
(g(U))),
and if m is taken sufficiently large, then
f
m
(g(U))
e
°ti.
Next let V be any
component of N(G) and suppose that V is wandering. Choose an infinite
sequence g, e G such that the sets g,(V) = V
t
are disjoint. Choose an integer n
such that f
n
(V) = U e %. As /" has finite degree, the collection {f
n
(Vt)}?
=1
must
contain an infinite number of components of N(G). However, for each / we see
that
(15) fW) =f"(g
i
(V)) =
4>t ° gt
°
for some $, e $(G). However, it is again easy to see from the precompactness of
the set 3>(G) that in fact the set
{<f>(si):
<f>
e
0>(G)}
is a finite collection of
components of N(G).
We next construct an example of an infinitely generated polynomial semigroup
with a wandering domain. It may be of some interest to note that our example
nevertheless satisfies the condition of being 'of finite type', to be given later, at
the beginning of §7.
Suppose that 0 <
a
<
§
and set
(16) p(z)
=
az(l+z).
Then 0 is an attracting fixed point of the mapping p. Let vv
0
= \ and
(17) w
n
=p"(w
0
).
Then w
n
are real, positive and
w
n
—» 0
as n
—>
°°.
Let r
0
= Aw
0
and s
0
=
\r
0
, where
A
is a positive constant to be determined later. Define
(18)
r
n
+ l
= ar
n
(l +
r
n
+2w
n
),
(19)
s
n+l
=as
n
(l+s
n
+2w
n
).
We first claim that the disks
(20) A
n
=
{z:
|z-wj «£/•„}
are all disjoint. Since the w
n
are real and decrease to 0, it is easily seen that it
suffices to show that
(21) r
n
+
r
n+1
<
\w
n
-
w
n+1
\.

372
A.
HINKKANEN
AND
G. J.
MARTIN
Now
\w
n
-
w
n+
i\
= w
n
(l -a- aw
n
) > \w
n
and
r
n
+
r
n+1
= r
n
{\ +a+ar
n
+ 2aw
n
) < 2r
n
.
Thus the disks will be disjoint
if
we can show
for
all
n
that
Ar
n
< w
n
. This is where
we must choose
A
suitably. We first observe that w
n
and r
n
eventually tend to zero
geometrically.
In
fact r
n + l
/r
n
^>a
and
w
n+1
/w
n
—>a
as
n
—>°°.
Therefore
the
following infinite product converges
and the
sequence
of
partial products
increases:
(22)
f[
(1
+
r
n
+
w
n
)
= a >
1.
We choose
A
with
A *£
1/(4a). Note that r
o
/w
o
=
A
< \.
We compute that
From this we deduce by induction that
-5
n
(1
+
r+w)<aAat5
w
n+
i
vv
oy=o
w
0
4
This establishes the first claim.
It
is clear that s,,
<
r
n
for
all n. We set
(23)
B
n
={z: \z-w
n
\<s
n
}^A
n
and claim that
(24) p(B
H
)<zB
H+l
.
To see this note that
if z =
w
n
+
5
n
t
with I'I
=
1>
then
= aw
n
{\
+
w
n
) + aw
n
s
n
t + as
n
t(l + w
n
+ s
n
t)
= w
n+l
+ tas
n
(l + 2w
n
+ s
n
t),
and hence
(25) \p(z)
-
w
n+1
\ ^as
n
(l
+s
n
+ 2w
n
)
=
s
n+i
.
Now the generators
of
our semigroup can be defined. For each
n ^
1
set
(z
-
wj
2
(26) q
n
{z)=
z
w
n
+
.
Then,
for
each
n, q
n
is a
quadratic polynomial whose Julia
set is the
circle
{z: \z
-
w
n
\ =
s
n
}.
As r
n
>s
n
,
there
is an
integer
m(n)
such that

SEMIGROUPS
OF
RATIONAL FUNCTIONS
373
Then set Po(z)
=p(z) and for n
2*
1
set
(28) Pn(z)
=
qZ
(n)
(z).
For
n^\, the
disk
B
n
is the
filled-in Julia
set of p
n
and the
complement
of the
disk
A
n
is
mapped into D.
It is
easy
to see
that
p
n
(D) c
D
for all n
3*0.
We
now
prove
the
following theorem.
THEOREM
5.2.
The
polynomial semigroup
G = (
Pn
(z):
n^O)
has
a
wandering domain.
Proof.
The
first thing
to
prove
is
that each
of the
disks
B
n
is a
component
of
the Fatou
set N(G) of
G.
To see
this, choose some
n
s*
1
and let g e G be an
arbitrary element. Then
g is
expressible
in
terms
of
the generators
as
(29)
g
=
Ph°Pi
2
°-°Pi
m
-
If
/
m
7*0,
n, then
p
im
(B
n
)<=:D.
This
is
because B
n
<=iA
n
and A
n
lies
in the
complement
of
A
im
=>
B
im
,
as the
disks
A
k
are
disjoint.
As
each
of the
generators
maps
D
into
itself,
we see
that g(B
n
)
c
D.
If i
m
=
n, then
p
im
{B
n
)
=
#„.
If i
m
=
0,
then Pi
m
(B
n
)
<=
2?
n+1
.
Thus every element
of G
either maps /?„ into
D,
maps
£„ to
itself,
or
maps
B
n
into
B
k
for
some
A:
>
n.
We
easily deduce that
G is
normal
on
£„.
As
dB
n
= /(/?„) c/(G),
we see
that
B
n
is
actually
a
component
of
N(G).
As
(30)
we find that each component
B
n
of
N(G)
is
wandering.
In
the
above example
we
have made some simplifications
to aid the
calculations. For instance, we used real parameters. However,
it is
not too difficult
to
see
that
the
construction
is
sufficiently robust
to
allow wandering domains
under more general circumstances. Notice that the attracting fixed point of p
0
is
in
the Julia
set of G and in the
boundary
of
N(G).
A
referee pointed
out
that
we
could replace the present
p
0
by
a
linear polynomial, but we have chosen
not to do
so
in
order
to
preserve
the
above extra property. Further examples arise
as
subsemigroups
of the
semigroup
G. For
instance,
we can
prove
the
following
result.
THEOREM
5.3. The polynomial semigroup
G
=
(p
2j
:
;
= 0,1,...)
has
the
property that the component
B
2
of the
Fatou
set is
wandering,
but
returns
to
the same component
infinitely
often.
Of course, this works since Pi{B
2
)
= B
2
for all n^l.
The point
is
that since
the
p
2
are distinct elements
of
G,
the
component
B
2
is a
wandering domain
but not a
strictly wandering domain.

374
A.
HINKKANEN
AND G. J.
MARTIN
6. Stable domains for
semigroups
We begin with a few definitions which should partly serve to classify the
dynamics of a rational semigroup on an invariant component of the Fatou set. Let
G be a rational semigroup and let U be a component of the Fatou set N(G) of G.
Recalling our notation that U
g
is the component of N(G) containing g(U), we
define the stabilizer of U to be
(31) Gu
=
{geG:
U
g
=U}.
Evidently G
a
is a subsemigroup of G. (Thus N(G)
<=
N(G
U
); equality might not
hold, and one can ask if U can be a proper subset of a component of N(Gu)-) If
Gy contains an element of degree 2 or more, we shall say that U is a stable basin
for G. Given a stable basin U for G, we say that it is
(i)
attracting
if U is a subdomain of an attracting basin of each g e G
v
with
(ii)
superattracting
if U is a subdomain of a superattracting basin of each
g e Gu with deggs*2;
(iii) parabolic if U is a subdomain of a parabolic basin of each g E
GU
with
gg;
(iv) 57ege/ if U is a subdomain of a Siegel disk of each g e G
v
with degg
2
s
2;
(v) Herman if i/ is a subdomain of a Herman ring of each g e G
L
, with
degg ^2.
In the classical case a stable basin must be one of the above types. We raise the
following two conjectures. Each is clearly true in the classical case, and in general
is a little weaker than the non-existence of wandering domains.
CONJECTURE
6.1. Let G be a (finitely generated) rational semigroup with
N(G)?
k
0. Then there is a stable basin U for G such that Gu has cofinite index
in G.
CONJECTURE
6.2. Let G be
a finitely generated rational
semigroup.
Then
for
each
component U of N(G), there is a stable basin V for G lying in
the
forward orbit of
U (that is, the set
{U
g
:
g e G}), such that G
v
has cofinite index in G.
If G
v
has cofinite index in G, we also say that V is a basin of cofinite index.
In the second conjecture we have included the assumption of finite generation
because we feel that the example of the previous section of a semigroup with a
wandering domain might be modified to give a counter-example to the more
general statement. We shall return to this later. Note however, that the example
given does not contradict the second conjecture even without the assumption of
finite generation, because the attracting basin of infinity lies in the forward orbit
of every component of the Fatou set and this basin has cofinite index in G (in
fact, it is equal to G). It might be interesting to give a criterion for when the
hoped-for cofinite index could be replaced by finite index. It would seem to us,
however, that the notion of finite index is perhaps not so useful.
EXAMPLE
4. Let h be a polynomial of degree at least 2 with distinct components
A and B of N(h) such that h(A)
=
h(B)
=
A and A contains the (super)attracting
fixed point a of h. Let g be a polynomial of degree at least 2 with distinct

SEMIGROUPS
OF RATIONAL FUNCTIONS 375
components U and V of N(g) such that g(U)
=
g(V)
=
U, U^B, V^A, and
aeV. There is m ^
1
such that h
m
(V)^V and
h
m
(U)^V.
Set / = h
m
and
G = </, g). Hence U and V are components of N(G). It is easy to see that
G
v
=
{f°F: fe G}.
Thus G
v
is of coindex 1 in G, while G
v
is not of finite index in G since
g"
°
f ° F e G for all F e G, n ^
1.
Furthermore, G
v
is not finitely generated even
if G is. For if G
v
= (g,,..., g
k
), then g, e G for all / so that g, =/
°
f) where
Fj
e G. But f
°
g" e G
v
for all n
5*
1, and not every f
°
g"
can lie in (g,, ..., g
k
).
There is, of course, a relationship between no-wandering-domains type results
and the existence of stable basins of cofinite index. We discuss this in the
following theorem.
THEOREM
6.1. Let G be a
rational semigroup
with no
wandering
domains.
Let U
be any component of
the
Fatou
set.
Then the forward orbit of U under G, that is,
{U
g
:
g E G}, contains a stable basin of
cofinite
index.
Proof.
Let G and U be as described above. Since U is not a wandering
component, the forward orbit of U is finite. Label the components of the forward
orbit Ui,
U
2
,...,
U
m
, with
\J^
=
IJ.
If for every j there is g
y
E G such that
gj(Uj)
c U
u
then G^ is easily seen to have cofinite index in G. Otherwise choose
k
s= 2
such that U\ does not lie in the forward orbit of V = U
k
. The number of
components of the forward orbit of V is therefore strictly less than that of U.
Proceeding by the obvious induction, we find a component W whose forward
orbit has fewest components and that W =
U
t
for some Ui^m. Then for every
component W
h
of the forward orbit of W there is g e G such that g(W
h
) c W, that
is,
W
hog
=
W, and it follows that W has cofinite index.
COROLLARY
6.1. Let G be a nearly abelian rational semigroup. Let U be any
component of the Fatou set. Then the forward orbit of U under G, {U
g
: g e G},
contains c stable basin of
cofinite
index.
In a sequel to this paper, we shall consider further results involving the concept
of a cofinite index. One of the simplest such results is the following: if
Gu = G = <gi,..., g
k
), then there is some / with
1 *s
/ ^ k such that 7(g,) n dU ^ 0.
We next discuss a few simple features of some stable basins for rational
semigroups.
First we point out that a stable basin can be attracting for a semigroup G, and
yet, there need not be a common attracting cycle fixed by each g E G. For
instance, let f(z)
=
z
2
+
c and g(z) = z
2
+
d, where c,d E
C\{0}.
If \c\, \d\ are
sufficiently small, then the disk D{\) of radius \ centred at 0 is mapped into the
disk D(\) by/", g" for some large n. Thus G = (/", g") is a polynomial semigroup
which contains {z:
\z\
<
2}
in its Fatou set. This disk contains the attracting cycles
for / and g (and hence for /" and g") and these are different if c ^
d.
Every h e G
maps D(\) into D(\) and is therefore (super)attracting. (We ask if it is possible to
show that no h E G is superattracting.) This can also happen for parabolic basins.
For instance, whenever there is a Mobius symmetry, we may produce a
semigroup by including the mapping and the symmetry. However, it seems
unlikely that this can happen for Siegel or Herman basins. It also seems unlikely

376 A. HINKKANEN AND G. J. MARTIN
that a basin can be superattracting with different superattracting fixed points
contained in the basin, unless they are related in some very special manner (cf.
Example 6 below, where two distinct superattracting fixed points are interchanged
by an element of 3>(G)).
In any case we are able to prove the following for nearly abelian rational
semigroups. Recall that by Theorem 4.1, if G is nearly abelian, then J(G) = J(g)
for all g e G of degree at least 2.
THEOREM 6.2. Let G be a nearly abelian rational semigroup and U a stable
component of N(G). Then U is attracting, superattracting, parabolic, Siegel or
Herman. In the Siegel case the basin U contains a single cycle fixed by each element
of G
a
. If U is of Siegel or Herman type, then Gy is abelian.
Proof.
We may replace G by G
a
. We denote the family of commutators of G
by 3>(G). We break the proof down into a case-by-case analysis. Now U is a
Herman ring for g
G
G if and only if U is doubly connected; otherwise U is simply
connected or of infinite connectivity (cf. [4, Theorem 7.5.3, p. 173, and Corollary
7.5.6, p. 174]). So we may now assume that U is not a Herman ring and consider
the remaining possibilities.
Case 1. Suppose that U is superattracting for g e G and let z
g
denote the
superattracting fixed point of g. That is, we suppose that U lies in a superattract-
ing basin for (g). Since g(U)
<=
U, the component U contains z
g
. We claim that
each h e G of degree at least 2 is superattracting in U.
Again we consider each case separately.
Case l(i). Suppose that U is attracting but not superattracting for h. As
h(U)^U,
the fixed point of h, say z
h
, lies in U. Let <£„,„, = [h
n
, g
m
]. Recall that
(f)
nm
is a Mbbius transformation and that, as h(U)^ U and g(U)<= U, we must
have
<f>
n
,
m
(U)
= U. We have
(32) h" °
g™
=
<!>„,„
og^on*.
The usual conjugate of g at z
g
is z -» z
d
for some d
2=
2, while the linearization of
h at
Zh
is of the form z
—*
Az with 0 < |A| < 1.
It may be that
(33) g
m
(z
h
) = z
g
for all large m. Suppose, for a while, that (33) does not hold for any m
5=
1. Let V
be a small topological open disk containing z
g
, to be specified in a moment. We
claim that if n
5=
n
0
, say, then there is a large m
0
depending on n such that
(g
m
o
h")(V) is multiply connected for all m^m
0
. This is clearly true if
z
g
£ (g
m
°h")(V) since g multiplies the 'argument' of the points in h
n
(V) by a
factor d (in suitable coordinates where z
g
has the role of the origin). Since the set
£/ ^ U*2»i 8~
k
(Zg) clusters to dU only and does not contain
Zh
(since (33) does not
hold),
the possibility that z
g
e (g
m
° h
n
)(V) can be avoided by taking n
0
large
enough and by choosing n^n
Q
, so that h
n
{V) must then lie in a fixed small
neighbourhood of
Zh-
Now
is also multiply connected since
(f>
n>m
is a homeomorphism.
So far there have been no restrictions on V. We now want to choose V so that

SEMIGROUPS
OF RATIONAL FUNCTIONS 377
(h
n
°
g
m
)(V) is simply connected for some n
2=
n
0
and for m corresponding to this
n as above. This then gives a contradiction, which proves that (33) must hold for
all large m.
Suppose first that none of the points h"(z
s
), for n 2=0, is a critical point of h.
Let V correspond to a sufficiently small round disk centred at the origin in the
coordinates where g becomes z^*z
d
. Then each g
m
(V) corresponds to such a
disk also and is simply connected. If V is small enough, then h is homeomorphic
in each set (h"~
]
°
g )(V) for n
2* 1
(since h has only finitely many critical points
in U, since the points h"(z
g
) tend to z
h
, and since h is a local homeomorphism at
Zh)-
Thus each set (h
n
°
g
m
)(V) is simply connected.
If h'(h
n
(z
g
)) = 0 for some n 2=0, then this holds for only finitely many n, all of
which are less than or equal to n
2
, say. So if v is fixed with v>n
2
, and if, for
some m 2*0, {h
v
°
g
m
){V) is simply connected and sufficiently small (which can be
arranged by taking V small enough), then also (h"
°
g
m
){V) is simply connected
for all n
2s
v and this same m. Let us choose here n > max{v, n
0
} and then choose
m depending on this n as before (so that (g
m
°
h")(V) is already known to be
multiply connected). The map (h
v
°
g
m
)(£ + z
g
) - h
v
(z
g
) behaves like £
k
for £ in a
neighbourhood of the origin, for some k
2= 2.
Thus z
g
has a topological disk
neighbourhood V for which (h
y
°
g
m
)(V) is simply connected, as required. This
completes the proof of (33).
Now fix m for which (33) holds. Set b
=
g
(d)
(z
g
)/d\
¥>
0 and note that g
U)
(z
g
)
=
0
for l^j^d-1. Suppose that (g
m
)
u)
(z
h
) = 0 for
1
=£/ ^
K
-1 and that
{g
m
)
(K)
{z
h
)lK\=a*0, where
K S* 1.
For k,n
s*
1, write
ip
k
,
n
=
g
m+k
°
h" and
Xk,n
=
h"°g
m+k
,
so that
ift
k
,n
=
<Pk,n°Xk,n
for some <p
M
e 0>(G). The chain rule
implies that
•
+dk
'\a\
nK
)
d
"
* 0
while (IAM)
0)
(Z/») = 0 for
1
^; <
K
d
k
. Similarly,
while
(Xk,n)
u)
(Zh)
=
0 for
1
«£/ < /cd*. Furthermore, we have
where a
kM
=
Xk,n(
z
h)
=
h"(z
g
) lies in a compact subset of
U.
Since the
(p
k>n
belong
to the precompact family 3>(G) of Mobius transformations, it follows that
\(<Pk,n)'(<x
kin
)\
s*
e >0 for all k,n
s*
1, for some e >0. This shows that h'(h
j
(z
8
)) * 0
for all /ssO. We have h'(z)—*
A
as z-+Zh- Since h'{z
g
)-*Zh as ;—>°c and since
0<
|A|
< 1, we obtain a contradiction by taking /: and n to be sufficiently large.
Case l(ii). Suppose that U is parabolic for h. Then the parabolic fixed point of
h lies on the boundary of U. Then for all m, we have (h
n
°g
m
)(z
g
)
=
h"(z
g
)—>
z
h
e dU. However, (g
m
° h
n
){z
g
) can be made to be arbitrarily close to z
g
by a
judicious choice of m, for a given n. As z
g
is an interior point of U and z>, is a
boundary point, we again reach a contradiction to the precompactness of the
family
{4>
n
J.
Case l(iii). Suppose that U is a Siegel disk for h. The orbit of z
g
under /i is
precompact
in U and its
closure
is
topologically
a
circle unless
h{z
g
) = z
g
.
Suppose
that h(z
g
)^z
g
. Choose a small neighbourhood V of
z
g
-
Since the preimages of z
g
under (the iterates of) g are isolated in U, we can find an n such that h"(z
g
) is not

378 A. H1NKKANEN AND G. J. MARTIN
a preimage of z
g
under g. Then, if V is small enough and m is sufficiently large,
the domain (h
n
° g
m
)(V) is simply connected, while (g
m
° h")(V) is not. Thus we
conclude that h(z
g
) = z
g
. Next, after conjugating by a suitable Mbbius transfor-
mation, we may assume that z
g
- z
h
= 0 and that the power series expansion of g
at the origin looks like z
d
+ higher order terms. As h has a Siegel disk centred at
0, its power series expansion looks like cz + higher order terms with c =
e
2m9
and
6 irrational. It follows that [g, h] is an elliptic Mobius transformation of infinite
order. Therefore U is a domain mapped onto itself by an elliptic Mobius
transformation of infinite order with an interior fixed point. Thus U is a round
disk. By the reflection principle applied to h in dU, we see that U cannot be the
Siegel disk of a map of degree at least 2. This completes the proof of Case 1.
Case 2. Suppose that U is attracting for g e G and let z
g
e V denote the
attracting fixed point of g. We claim that U is attracting for each h G G of degree
at least 2.
Case 2(i). Suppose that U is superattracting for h. This case is covered in Case
l(i).
Case 2(ii). Suppose that U is parabolic for h. Then the proof given in Case l(ii)
easily generalises.
Case 2(iii). Suppose that U is a Siegel disk for h. As U is a component of
N(G) = N(g), we find that c
g
e U for some critical point c
g
of g. The orbit of c
g
under h is again precompact and its closure is topologically a circle, unless
h{c
g
) = c
g
. If h(c
g
)
T*
c
g
, then h" ° g has degree at least 2 near c
g
, and g ° h" does
not, for a suitable choice of n. This leads to a contradiction. Thus h(c
g
) = c
g
.
Similarly, there is c' e U with c' ¥^c
g
and g(c') = c
g
. Now g
2
is branched over c',
so that h would have to fix both c
g
and c'. This is impossible.
Case 3. Suppose that U is parabolic for g e G and let z
g
denote the parabolic
fixed point of g. As above, it is clear that z
g
e dU. We claim that U is parabolic
for each h £ G of degree at least 2.
Case 3(i). Suppose that U is attracting or superattracting for h. This case is
covered in Cases l(ii) and 2(ii).
Case 3(ii). Suppose that U is a Siegel disk for h. As above, a critical point c
g
of
g lies in U. Also there is c' G
U\{C
8
}
with g(c') = c
g
. The argument concerning
the branching leads, as before, to a contradiction since it implies that c
g
and c' are
both fixed by h.
Suppose that U is of Siegel or Herman type, so that U can be mapped by a
conformal mapping
<p
onto a disk D or to a non-degenerate annulus A, each
centred at the origin. Let g,h G G = Gu map U (conformally) onto
itself,
so that,
in the Siegel disk case, g and h fix the same point a of U, as we proved above,
and we may assume that then <p(a) = 0. It is easily seen that
<p~
]
°
g °
<p
and
(p~
x
o
h o
(p
are berth rotations (that fix each boundary component of A, as a set, in
the Herman ring case), so that these rotations commute. Thus g and h also
commute (in U, and hence globally), as claimed. This proves the last statement of
Theorem 6.2, and so the proof of Theorem 6.2 is complete.
We now give a few examples to show that the results and description contained
in the proof of the above theorem are the best possible.
EXAMPLE 5. Common parabolic basins. Set f(z) = z
2
-\ and g = -f. The
semigroup G = (/, g) was shown to be nearly abelian just before the statement of

SEMIGROUPS
OF RATIONAL FUNCTIONS 379
Theorem 4.3. Now/has a parabolic cycle at z = -\ and g at z =
2-
Thus a nearly
abelian semigroup can have different parabolic cycles in the same stable basin.
More precisely, there is a component U of N(G) containing the origin such that
each of/
2
and g
2
maps U onto itself and has a parabolic fixed point on dU, the
fixed point being -\ for/
2
and 5 for g
2
.
EXAMPLE
6. Common superattracting basins. Set f(z)
=
(z
2
- c
2
)
2
+ c and
g(z) = -c - (z
2
- c
2
)
2
. Then c is a superattracting fixed point for / and -c is a
superattracting fixed point for g. If #(z) = -z then g(z) =
<£
°/ It is easy to see
that
<f>(J(g)) =
J(g) and
<t>(J(f)) =
J(f) and hence that </ g) is a nearly abelian
polynomial semigroup. If \c\ is small enough, then both / and g map the disk
\z- |z| <
2}
into
itself,
and thus / and g have a common superattracting basin.
Notice that the proof of the classification of stable basins in the nearly abelian
case implies that the degree of/and g is greater than 2 if they are to have distinct
superattracting fixed points (since each of /and g must map the fixed points of the
other onto their own fixed points).
EXAMPLE
7. Set/(z) = z/(l
+
z - z
2
) and g(z)
=
\z
+
z
2
where 0<
A
<
1.
Then
J(g) is a Jordan curve while /(/) is a Cantor subset of the real line. The mapping
/has a parabolic fixed point at 0, N(f) is connected, each of the upper and lower
half planes is completely invariant under / and there is e > 0 such that the
interval (0, e) g N(f) fl N(g) because /((0, e)) c (0, e) and g((0, e)) c (0, e). Let
G be the semigroup generated by /and g, that is, G = </, g). (Of course, this G is
not nearly abelian.) Then each /ieG has an attracting or parabolic fixed point at
0. If e is small enough and we set B =
{z:
\z -
e\
< e), then f(B) ci B (to see this
note that l/f(l/z) = z + l-l/z). We claim that g(B)^B. This is not immedi-
ately apparent, but it is a calculation which we leave to the reader to verify.
(It suffices to choose any e such that 0< e <
2(1
- A) and 4e(l-A-e)<
(A + 2e)(l-
K-2E).)
It follows that BgN(C)^0 and therefore that Oe
dN(G).
Thus the stable basin for G containing B is both parabolic and attracting
and contains the parabolic/attracting fixed point in its boundary.
To conclude this section, we offer a small preview of our sequel concerning the
structure of stable basins and the attractors within. Note that if U is a stable basin,
then, in general, each g
G
G maps this basin into
itself.
As a motivational
example, suppose that G is finitely generated. Then the generators map U into
itself;
for simplicity, suppose that the generators are (super)attracting in U. We
can then think of G as giving rise to a family of contractions (in the hyperbolic
metric, say). Using the standard theory, one can construct a minimal forward
invariant subset of the stable basin U. This is the attractor of the basin. Under
suitable circumstances, it will be a fractal, and one can consider properties of
measures on the attractor, related to the dynamics of G in the stable basin.
7.
Polynomial semigroups
We begin with a definition of a finiteness condition for polynomial semigroups
that will imply at least some nice properties related to the structure of the filled-in
Julia set. We say that a polynomial semigroup G is of finite type if G satisfies the
following two conditions.

380
A. H1NKKANEN AND G. J. MARTIN
(1) For any positive integer N there are only finitely many polynomials in G
whose degree is less than N.
(2) There is a domain D in C, whose complement C \ D is a bounded
continuum, such that each g e G maps D into
itself,
that is g(D) c D.
Similarly one could define a rational semigroup G, which is not a polynomial
semigroup, to be of finite type if the analogue of Condition (1) holds, that is, if for
each n
5*
1, there are only finitely many elements of G whose degree does not
exceed n. However, we shall not study non-polynomial semigroups of finite type
further in this paper.
We make a few remarks to clarify the utility of this definition for polynomial
semigroups. The finiteness condition is immediate if G is finitely generated by
elements of degree at least 2, while if G is not finitely generated, the degrees of
the generators must increase. In particular, there are only finitely many
generators of any given degree. The domain D can be thought of as part of the
attracting basin of ». The condition guaranteeing the existence of D need only be
checked on the generators of the semigroup. If G is finitely generated then it is
relatively clear that the complement of a sufficiently large disk suffices as D.
Finally observe that every element of G of degree 1 must be an elliptic Mobius
transformation of finite order, since otherwise there would be infinitely many
elements of degree 1. Also the group generated by this family of elliptic
transformations must be finite and, as each element must fix the point infinity, this
group is cyclic and therefore generated by a single elliptic Mobius transformation.
THEOREM
7.1. If G is a finitely generated polynomial semigroup with at most
finitely many elements of
degree
1, then G is of finite type.
For a polynomial h we denote ihefilled-in Julia set of h by K(h). Recall that
(34) K(h)
= {z
G
C: {h
n
(z): n = 1,
2,...}
is bounded}.
For a polynomial semigroup G, let K(G) be the closure of that set of points
zeC such that G(z)
=
{g(z): g e G} has a finite limit point. We call K(G) the
filled-in Julia set of G. It is clear that K(g) c K(G) for all g e G and that K(G) is
backward invariant.
The main result we offer here is the following.
THEOREM
7.2. Let G be a polynomial semigroup of finite type. Then there is a
domain V
=>
D, containing a neighbourhood of °°, such that V coincides with the
unbounded component of
the
complement of
the
set
(35)
and has the following
property:
for any z £ V (and hence any z e D) and for any
compact subset K ofC,
there
are only finitely many g e G such that g(z) e K, and,

SEMIGROUPS
OF
RATIONAL FUNCTIONS
381
furthermore,
V is
the largest domain containing the point
at
infinity that has this
property.
Proof.
First
let F be the
smallest closed
set
containing U/ieC^C
1
)
an
^
such
that g~\F)^F
for all g e
G. Clearly F
£
K(G)
as K(G) is
closed
and
backward
invariant. Suppose now that
the
complement
of
F in
C
contains
a
neighbourhood
of infinity. Then
let V
denote
the
unique unbounded component
of the
complement
of Fin C.
Clearly C\Fcontains
A(G) = C\ K(G).
Notice that
z e A(G) if for all w in
some neighbourhood
of z, the set G(w)
clusters only
to
°°. (The
set A(G)
should
be
thought
of as the
attracting basin
of
infinity.) We shall next prove that
for
every
z e V,
the
set G(z)
clusters only
at
°°.
Since
V is
open, this then shows that V^A(G),
so
that
A(G)
=
C\K(G)
contains
a
neighbourhood
of
infinity
and
therefore
has a
unique unbounded
component
W in C. It
also follows that
V c
W.
But
since
C\K(G)
=
A(G)
c
C
\
F,
as
observed earlier,
it
follows,
on the
other hand, that W
£
V. Hence
V =
W.
Let
us
recall
the
assumption that there
is a
planar domain
D
whose
complement
is a
continuum
K
such that every
g s G
maps
D
into
itself.
This
implies,
of
course, that
G is
normal
in D and so D
<=N(G).
It is
easily seen that
K(g)
<=
K for all g e G of
degree
at
least
2.
Furthermore,
we
clearly have
g~
x
(K)^K
for all g e
G. Since
K is
closed,
the set F
lies
in K and so C\F
contains D. Hence
C\F
contains
a
neighbourhood
of
infinity,
as we
assumed
at
the beginning
of
the
proof.
It
also follows that
V
contains D. This then completes
the proof
of
Theorem
7.2.
It remains
to
prove that
for
every
z e V, the set G(z)
clusters only
at
°°. To
do
this,
pick
z
G
V.
For
each geGof degree
at
least 2,
let S
g
(z)
denote
the
Green's
function
of
the complement
of
K(g) with pole
at
infinity. For each
g e
G,
z
lies
in
the complement
of
K(g)
and so
S
g
(z) >0. Next, we have
(36) S
g
(g(z))
=
(degg)S
g
(z)
for z e
C
\
K(g).
(This well known
but
we recall
the
proof.
Both sides behave like
(degg)log|z|
+
O(l)
near infinity,
so
their difference
is a
bounded harmonic function outside
of
K(g).
As
g
maps
the
boundary
of K(g)
onto
itself,
both sides vanish
at the
boundary.
Hence
the
difference vanishes identically.)
Returning
to
our
proof,
from consideration
of
the associated Robin constants
we
can
see that
the
logarithmic capacity
of
K(g) satisfies Cap(/C(g))~
1
= M
l/{n
'
])
,
where
n
= deg
g
and M is the
modulus
of the
leading coefficient
of g. As
V contains
a
neighbourhood
of
infinity,
we may
consider
the
Green's function
T(z)
of V
with pole
at
infinity. Let
g s G be
any rational function
of
degree
2 or
more.
As V is
contained
in the
complement
of K(g) and as
both
T and S
g
have logarithmic singularities
at °°, we see
that
for
each
g s G the
function
T(z)
—
S
g
(z)
is
bounded
and
harmonic
in V,
non-positive
on the
boundary
of
V and hence non-positive
on
V. Thus S
g
(z)
^ T(z).
Hence,
for
each
z e V,
(37)
mf{S
g
(z):
gsG}**T(z)>0.
Let
N
s*
1.
Now,
by
assumption, there
are
only finitely many
g e G of
degree
less than N. Therefore the numbers S
g
(g(z)) = (degg)Sg(z)
s*
(degg)T(z) tend
to
infinity.
We
want
to
show that this implies that
g(z)
also tends
to ».
Now,
for

382
A.
HINKKANEN AND
G. J.
MARTIN
each
g
e
G of
degree
at
least
2, the set K(g) is
compact
and
lies
in
a
fixed disk
of
radius
R
centred
at the
origin (here
R
depends
on V
only).
The
set
K(g)
has
positive logarithmic capacity,
so c
= -\og(Cap(K(g)))
is
well defined. There
is a
probability measure
m on K(g)
such that
(38)
S
g
(z)-c=\ \og\z-w\dm(w).
Suppose that
\g(z
)|
<
r.
If
Cap(/C(g))
>
L
>
0, then
S,(g(z))<-\og(L)
+
\og(R +
r).
Thus
the
result
we are
after follows unless there
is no
positive lower bound
on
Cap(/C(g))
for g
E
G.
Now
z is at a
fixed positive distance
d
from every /C(g).
For
a
given large
r,
consider
two
disjoint subsets
of
elements
of
G.
First consider those
g e G for
which Cap(K(g))>
\d and
|g(z)|<r.
We
see as
above that there
can
only
be
finitely many such
g.
Next, consider those
g e
G for
which
\g(z)\
< r
and
Cap(/C(g))
=s
\d.
Let
g
be
a
member
of
this latter
set and set
n
=
degg
and
L
=
Cap(/C(g)). Then
(39)
log|s
-w\
dm(w)^\ogd,
while since |g(z)|<r,
we
have
(40)
[
log \g(z)
-
w\
dm(w) < \og(R
+ r).
J
K(g)
Since S
g
(g(z))
=
M5
S
(Z),
we see
that
\og(R
+
r)
-
log(L) >*(log(<0
-
log(L));
hence
\og{R
+
r)>n\ogd +
(n-
l)(-log
L)
^n\ogd +
{n
~\){-\og\d)
= n log
2
+
log
jd.
For
a
given
r
this last equation implies
an
upper bound
for n, say n <n(r,
d,
R).
By assumption, there
are
only finitely many
g e
G
whose degree
is at
most
n(r,
d, R),
and so in
this second class there
are at
most finitely many
g e
G.
Hence
for
any large
r
there
are
only finitely many
g
e
G for
which
\g{z)\
<
r.
Thus
Theorem
7.2 is
proved.
As
a
final example
of
relevance
to the
above discussion
of
the filled
in
Julia
set,
consider
the
subgroup
of
SL(2, C) generated
by the
matrices
Then
A
and
B
represent parabolic Mobius transformations.
It is of
great interest
to study
the set
of z
2
e
C
for
which
the
group (A,
B)
is
discrete. (Note that
if z
gives rise
to a
certain
B
then
—z
gives rise
to
B~\
so
that
z
and
— z correspond
to
the same group (A, B). Thus
it is
natural
to
parametrize
(A, B) by
means
of
z
2
-)
This
set is
known
as the
Riley slice through Schottky space.
The
classical
J0rgensen's inequality
[10]
implies that
for
discrete non-elementary groups,
we

SEMIGROUPS OF RATIONAL FUNCTIONS 383
have
\z\ s* 1.
For a given X e (A, B), we denote by
p
x
{z
2
)
the polynomial in z
2
given by the square of the A^-entry of the matrix X (it is obvious that this gives
a polynomial in z, but, in fact, by [8], we get a polynomial in z
2
). The family of
polynomials
& =
{p
x
: Xe(A,B)} is closed under composition (this can be
deduced using [8, first comment on p. 211, and Theorem 7.13, p. 212]) and so
forms a polynomial semigroup. Any polynomial p
x
+ 2 is the trace of a
commutator in the group (A, B). Notice that z
2
e 9 (since X = [A, B]). In a
geometrically finite discrete group, the set of traces of commutators is discrete
(there are more general relevant results that need not concern us here). Since
SF
is a polynomial semigroup,
\z\
<
1
implies that the points z
2
"
+ 2,
for n 3*0, form a
set of commutator traces with a finite accumulation point. This gives J0rgensen's
inequality. More generally, one can show that (A, B) is discrete if, and only if, the
orbit of z
2
under the polynomial semigroup
2F
has no finite accumulation points.
This,
in turn, holds if, and only if, z
2
does not lie in the filled-in Julia set K{cF) of
the polynomial semigroup
SF
minus the points with finite orbit (these correspond
to very special discrete groups, such as hyperbolic two-bridge-knot complements,
including the figure-eight knot, for example). It turns out that K(&) is the
complement of the space of discrete and free groups generated by two parabolics.
Equivalently, this subset of the plane is a model of the moduli space for the
four-times-punctured sphere.
It turns out that this setting is fairly typical. That is, the moduli space M of
discrete groups free on two generators (of given but fixed trace) is a one-complex-
dimensional space which is modelled by the complement of the filled-in Julia set
of some polynomial semigroup (constructed from generalized trace polynomials).
We know from Teichmuller theory that M
U {<»}
is simply connected. It would be
nice to know if the boundary of this space is topologically a circle. One can
deduce various things from the dynamics of the polynomial semigroup, but at
present, we seem far from being able to get the sort of results that we would like.
Finally, here is another hoped-for result.
CONJECTURE 7.1. If G is a nearly
abelian
polynomial semigroup, then there is a
neighbourhood of
°°
on which G is analytically conjugate into the nearly abelian
semigroup T
=
(z-*az
n
:
\a\
= 1, n = 1, 2, 3,...).
References
1. I. N.
BAKER,
'Repulsive fixed points of entire functions', Math. Z. 104 (1968) 252-256.
2.
A.
BEARDON,
'Symmetries of Julia sets',
Bull.
London Math. Soc. 22 (1990) 576-582.
3.
A.
BEARDON,
'Polynomials with identical Julia sets', Complex Variables Theory Appl. 17 (1992)
195-205.
4.
A.
BEARDON,
Iteration
of
rational
functions (Springer, New York, 1992).
5.
C.
CARATHEODORY,
Theory of functions of
a
complex
variable,
vol. II (Birkhauser, Basel, 1950;
Chelsea, New York, 1981).
6. P.
FATOU,
'Sur les gquationes fonctionnelles',
Bull.
Soc. Math. France 47 (1919) 161-271, 48
(1920) 33-94, 208-314.
7. F. W.
GEHRING
and G. J.
MARTIN,
'Iteration theory and inequalities for Kleinian groups',
Bull.
Amer. Math. Soc. 21 (1989) 57-63.
8. F. W.
GEHRING
and G. J.
MARTIN,
'Commutators, collars and the geometry of Mobius groups', J.
Analyse Math. 63 (1994) 175-219.
9. A.
HINKKANEN
and G. J.
MARTIN,
'Julia sets of rational semigroups', Math. Z. to appear.
10.
T.
JORGENSEN,
'On discrete groups of Mobius transformations', Amer. J. Math. 98 (1976) 739-749.
11.
G.
JULIA,
'Memoire sur l'iteration des substitutions rationnelles', J. Math. Pures Appl. 8 (1918)
47-245.

384
SEMIGROUPS OF RATIONAL FUNCTIONS
12.
B.
MASKIT,
Kleinian
groups (Springer,
New
York, 1987).
13. CH. POMMERENKE,
'Uniformly perfect sets
and the
Poincare metric', Arch. Math.
32
(1979)
192-199.
14.
J. L.
SCHIFF,
Normal families
(Springer,
New
York, 1993).
15.
D.
SULLIVAN,
'Quasiconformal homeomorphisms
and
dynamics
I:
Solution
of the
Fatou-Julia
problem
on
wandering domains', Ann.
of
Math.
122
(1985) 401-418.
Department
of
Mathematics
University
of
Illinois
at Urbana-Champaign
273 Altgeld Hall, MC'-382
1409 West Green Street
Urbana
Illinois
61801
U.S.A.
Department
of
Mathematics
and Statistics
The University
of
Auckland
Private
Bag
92019
Auckland
New Zealand
and
Centre
for
Mathematics
&
Applications
Australian National University
Canberra
ACT
Australia
2601