Content uploaded by Mark Shattuck
Author content
All content in this area was uploaded by Mark Shattuck on Jan 04, 2023
Content may be subject to copyright.
ON A CONJECTURE OF LIN AND KIM CONCERNING A
REFINEMENT OF SCHR ¨
ODER NUMBERS
TOUFIK MANSOUR AND MARK SHATTUCK
Abstract. In this paper, we compute the distribution of the first letter statistic on nine
avoidance classes of permutations corresponding to two pairs of patterns of length four. In
particular, we show that the distribution is the same for each class and is given by the entries
of a new Schr¨oder number triangle. This answers in the affirmative a recent conjecture of
Lin and Kim. We employ a variety of techniques to prove our results, including generating
trees, direct bijections and the kernel method. For the latter, we make use of in a creative
way what we are trying to show to aid in solving a system of functional equations satisfied
by the associated generating functions in three cases.
1. Introduction
Given two permutations π=π1···πn∈ Snand τ=τ1···τk∈ Sk, we say that πcontains the
pattern τif there exist indices i1< i2<··· < iksuch that πi1···πikis order isomorphic to τ,
that is, πia> πibif and only if τa> τb. Otherwise, πis said to avoid the pattern τ. Moreover,
we say that πavoids a set Lof patterns if it avoids each pattern in L, and let Sn(L) denote
the subset of Snwhose members avoid L. In recent decades, the study of pattern avoidance
in permutations has been the object of considerable attention (see, e.g., [7] and references
contained therein).
An inversion within a permutation σ=σ1···σn∈ Snis an ordered pair (a, b) such that
1≤a<b≤nand σa> σb. The inversion sequence of σis given by a1··· an, where
airecords the number of entries of σto the right of iand less than ifor each i∈[n].
The systematic study of patterns in inversion sequences was initiated only relatively recently
in [4, 13]. Analogous problems, such as avoidance of vincular [12] or multiple [20] patterns,
have been considered on inversion sequences in parallel to those on permutations represented
in the one-line notation.
The (large) Schr¨oder number Sn(see [17, A006318]) is defined recursively by
nSn= 3(2n−3)Sn−1−(n−3)Sn−2, n ≥3,
with S1= 1 and S2= 2, and arises as the enumerator of several avoidance classes of per-
mutations corresponding to a pair of patterns of length four. In particular, combining the
results from [5, 8, 18], one has the Snenumerates Sn(σ, τ) for the following ten inequivalent
pairs (σ, τ):
I. (1234,2134) II. (1324,2314) III. (1342,2341) IV. (3124,3214) V. (3142,3214)
VI. (3412,3421) VII. (1324,2134) VIII. (3124,2314) IX. (2134,3124) X. (2413,3142).
Date: January 3, 2023.
2010 Mathematics Subject Classification. 05A15, 05A05.
Key words and phrases. pattern avoidance, combinatorial statistic, kernel method.
1
2 T. MANSOUR AND M. SHATTUCK
This answered in the affirmative a conjecture originally posed by Stanley (see [8] for details).
Moreover, outside of symmetry, there are no other such pairs (σ, τ) for which |Sn(σ, τ)|=Sn.
In this paper, we obtain a refinement of this result in several cases by considering distributions
of certain statistics on the various classes. For other refinements of the Schr¨oder numbers,
see, e.g., [2, 14–16].
n\k123456
1 1
2 1 1
3 2 2 2
4 4 6 6 6
5 8 16 22 22 22
6 16 40 68 90 90 90
Table 1. The new Schr¨oder triangle Sn,k for 1 ≤k≤n≤6.
In a recent paper, Lin and Kim [11] introduced a new triangle Sn,k for Schr¨oder numbers in
their study of inversion sequences; see Table 1 above. Here, we find it here more convenient
for what follows to start from k= 1 instead of k= 0, as was done in [11]. Note that Sn,k is
given recursively by
Sn,k =Sn,k−1+ 2Sn−1,k −Sn−1,k−1,1≤k≤n−2,
with Sn,n =Sn,n−1=Sn,n−2for n≥3 and S1,1=S2,1=S2,2= 1. Lin and Kim showed that
Sn,k enumerates the inversion sequences π1···πn∈ {In(021) |πn≡kmod n}. Then they
state the following conjecture which provides a connection between inversion sequences and
pattern avoidance in permutations.
Conjecture 1. (Lin and Kim [11].) Let (ν, µ) be a pair of patterns of length four. Then
Sn,k =|{σ1···σn∈ Sn(ν, µ)|σn=k}|
for all 1 ≤k≤nif and only if (ν, µ) is one of the following nine pairs:
(4321,3421),(3241,2341),(2431,2341),(4231,3241),(4231,2431),
(4231,3421),(2431,3241),(3421,2431),(3421,3241).
Further, it was also shown in [11] that S(n, k) for 1 ≤k≤ngives the cardinality of the
restricted class of inversion sequences {π1···πn∈In(≥,−, >) : πn=k−1}. For other recent
results concerning the avoidance of a pattern of relation triples by inversion sequences, see,
e.g., [1, 3, 9, 10].
Given 1 ≤i≤n, let Sn,i(σ, τ ) denote the set of permutations of length navoiding σand τand
starting with i. Note that one may consider equivalently the distribution of the first letter
statistic on the set of permutations avoiding the reversal of the two patterns in question in
each case. Here, we confirm the conjecture by showing that |Sn,i(σ, τ)|=Sn,i for each of the
nine pairs above (where the patterns in each pair are reversed). It is seen that these nine cases
are derived from only six of the ten symmetry classes (I)-(X) above. It should be remarked
that the reversal, complement and inverse operations do not respect the first letter statistic
and thus members of the same symmetry class do not have the same first letter distribution
in general. Moreover, these are the only nine pairs such that |Sn,i(σ, τ )|=Sn,i for all i; see
Table 2 below which rules out all other possible pairs (σ, τ) for which |Sn(σ, τ)|=Sn.
ON A CONJECTURE OF LIN AND KIM 3
In several cases, we will make use of a generating function approach to establish the result.
Note that by the kernel method [6], one can show
(1)
X
n≥1 n
X
k=1
Sn,kyk!xn=xy(2 −3x−3y+ 3xy) + xy(x+y−xy)(xy +p1−6xy +x2y2)
2(1 −2x−y+xy),
which reduces when y= 1 to the well-known formula for the Schr¨oder number generating
function given by
(2) X
n≥1
Snxn=1−x−√1−6x+x2
2.
σ, τ {|{π∈S8(σ, τ )|π1=k}|}8
k=1 σ, τ {|{π∈S8(σ, τ )|π1=k}|}8
k=1
2413,4123 1584,1036,996,956,879,751,533,233 3142,4123 1584,1584,1252,912,637,443,323,233
3142,3214 1584,1584,736,396,292,304,488,1584 2341,2413 1584,488,304,292,396,736,1584,1584
2341,3142 1584,811,587,489,481,577,855,1584 2413,3214 1584,855,577,481,489,587,811,1584
2431,3421 1806,1022,710,614,644,795,1161,1806 2431,4231 1806,1092,1008,1045,1120,1134,924,429
2314,3124 1806,1092,752,629,629,752,1092,1806 2314,3214 1806,1092,752,629,629,752,1092,1806
2341,3241 1806,1092,752,629,629,752,1092,1806 2413,3142 1806,1092,752,629,629,752,1092,1806
2431,3241 1806,1092,752,629,629,752,1092,1806 2134,3124 1806,1161,795,644,614,710,1022,1806
3241,3421 1806,1806,1198,678,406,342,516,1806 3214,3241 1806,1806,1220,672,390,342,516,1806
3124,4123 1806,1806,1502,1152,840,594,429,429 3214,4213 1806,1806,1502,1152,840,594,429,429
3241,4231 1806,1806,1502,1152,840,594,429,429 3412,4312 1806,1806,1502,1152,840,594,429,429
3421,4231 1806,1806,1502,1152,840,594,429,429 3421,4321 1806,1806,1502,1152,840,594,429,429
4123,4132 1806,1806,1806,1412,928,512,224,64 4123,4213 1806,1806,1806,1412,928,512,224,64
4132,4213 1806,1806,1806,1412,928,512,224,64 4132,4231 1806,1806,1806,1412,928,512,224,64
4132,4312 1806,1806,1806,1412,928,512,224,64 4213,4231 1806,1806,1806,1412,928,512,224,64
4213,4312 1806,1806,1806,1412,928,512,224,64 4231,4312 1806,1806,1806,1412,928,512,224,64
4312,4321 1806,1806,1806,1412,928,512,224,64 3124,3214 1806,1806,788,540,484,540,788,1806
3412,3421 1806,1806,788,540,484,540,788,1806 2314,2341 1806,516,342,390,672,1220,1806,1806
2134,2314 1806,516,342,406,678,1198,1806,1806 2134,2143 1806,788,540,484,540,788,1806,1806
2341,2431 1806,788,540,484,540,788,1806,1806 1432,2413 233,323,443,637,912,1252,1584,1584
1432,3142 233,533,751,879,956,996,1036,1584 1234,2134 429,429,594,840,1152,1502,1806,1806
1243,2143 429,429,594,840,1152,1502,1806,1806 1324,2134 429,429,594,840,1152,1502,1806,1806
1324,2314 429,429,594,840,1152,1502,1806,1806 1342,2341 429,429,594,840,1152,1502,1806,1806
1432,2431 429,429,594,840,1152,1502,1806,1806 1324,3124 429,924,1134,1120,1045,1008,1092,1806
1423,4123 429,924,1344,1582,1582,1344,924,429 1432,4132 429,924,1344,1582,1582,1344,924,429
1234,1243 64,224,512,928,1412,1806,1806,1806 1243,1324 64,224,512,928,1412,1806,1806,1806
1243,1342 64,224,512,928,1412,1806,1806,1806 1243,1423 64,224,512,928,1412,1806,1806,1806
1324,1342 64,224,512,928,1412,1806,1806,1806 1324,1423 64,224,512,928,1412,1806,1806,1806
1342,1423 64,224,512,928,1412,1806,1806,1806 1342,1432 64,224,512,928,1412,1806,1806,1806
1423,1432 64,224,512,928,1412,1806,1806,1806
Table 2. All symmetries of classes I-X according to first letter statistic. Note
the 9 boldface cases corresponding to the conjecture and the 9 italicized cases
obtained by complementation.
This paper is organized as follows. In the next section, we show six cases of the conjecture
above using various methods such as induction, bijections and generating trees. In the third
section, we prove the remaining three cases, each of which involves 1243 and another pattern,
4 T. MANSOUR AND M. SHATTUCK
by considering the joint distribution of the first and second letter statistics. This permits one
to write a system of recurrence relations in each case which may then be expressed in terms
of some auxiliary generating functions leading to a system of functional equations. At this
point, one can use the conjecture itself in these particular cases along with the kernel method
to ascertain a potential solution to the aforementioned system, which may then be shown to
be the actual solution. Taking the variable that marks the second letter statistic to be unity
then recovers formula (1) and demonstrates the desired equality of distributions.
We remark further that in order to prove the final three cases we make use of a special
decomposition of the respective generating functions into what we term positive and negative
parts as well as parts corresponding to certain ordered pair values of the parameters in
question (see, for example, the various generating functions that are defined following the
proof of Lemma 7 below). Such a decomposition is necessary in translating the system of
recurrences at hand into a system of functional equations. In addition, the decompositions
of the permutations themselves required in deriving the recurrences for the last three cases
involving 1243 (see proofs of Lemmas 7, 11 and 16) are apparently new. Finally, in establishing
the recurrences in the cases (1324,1423) and (1342,1423) (see Lemma 4), we make use of
generating trees in not only tracking the labels of offspring but also the value of the first
letter statistic in the offspring. We do not know of other examples of generating trees being
used in this way in conjunction with a statistic auxiliary to the tree structure such as the one
recording the first letter.
In several instances, the distribution of the first letter statistic on the pattern pair in question
follows as a special case of a more general distribution. For one’s reference, listed below are
the places within the paper where the specific cases are shown.
Pattern Pair Reference Pattern Pair Reference Pattern Pair Reference
1234,1243 Theorem 2 1243,1324 Corollary 20 1243,1342 Corollary 15
1243,1423 Corollary 10 1324,1342 Theorem 2 1324,1423 Theorem 6
1342,1423 Theorem 6 1342,1432 Theorem 3 1423,1432 Theorem 2
Table 3. Places where specific cases of Conjecture 1 are proven.
2. Pattern avoidance and the new Schr¨
oder triangle
In this section, we enumerate members of Sn,i(σ, τ ) confirming the conjecture in six of the
cases, the first three of which we treat together in the following result.
Theorem 2. If n≥1and 1≤i≤n, then |Sn,i(σ, τ)|=Sn,i for (σ, τ) = (1234,1243),
(1324,1342) and (1423,1432).
Proof. We prove the case when (σ, τ) = (1234,1243), the others being similar. Let An,i =
Sn,i(1234,1243) and we first write a recurrence for an,i =|An,i|. Note that members of An,i
where 1 ≤i≤n−3 must have second letter `∈[i−1] or `=n−1, n, for otherwise there
would be an occurrence of 1234 or 1243 starting with i`. Furthermore, the first letter iis
seen to be extraneous concerning the avoidance of 1234 or 1243 if `∈[i−1] and thus may be
deleted. Similarly, `may be deleted in cases when `=n−1 or n. This implies the recurrence
an,i = 2an−1,i +
i−1
X
`=1
an−1,`,1≤i≤n−2,
ON A CONJECTURE OF LIN AND KIM 5
with an,n =an,n−1=Pn−1
i=1 an−1,i for n≥3.
Before proceeding further, note that
Sn,n−2=
n−2
X
k=1
(Sn,k −Sn,k−1) =
n−2
X
k=1
(2Sn−1,k −Sn−1,k−1)=2Sn−1,n−2+
n−3
X
k=1
Sn−1,k
=
n−1
X
k=1
Sn−1,k.
Thus, the an,i and Sn,i are both given by the sum of the entries of the previous row if
i∈[n−2, n], with the two arrays also agreeing for n= 1,2. Therefore, to complete the proof,
it suffices to show
(3) Sn,i = 2Sn−1,i +
i−1
X
`=1
Sn−1,`,1≤i≤n−2.
To do so, we proceed by induction on nand i, the i= 1 case clearly holding since Sn,1=
2Sn−1,1for all n≥3. So assume n≥4 and 2 ≤i≤n−2. Note further that (3) also
holds when i=n−1 for n≥3, which follows from the work above. Then by the induction
hypothesis, we have
Sn,i =Sn,i−1+ 2Sn−1,i −Sn−1,i−1= 2Sn−1,i−1+
i−2
X
`=1
Sn−1,` + 4Sn−2,i + 2
i−1
X
`=1
Sn−2,`
−2Sn−2,i−1−
i−2
X
`=1
Sn−2,`
= 2Sn−1,i−1+ 4Sn−2,i +
i−2
X
`=1
(Sn−1,` +Sn−2,`).(4)
Upon substituting (3) into (4), and simplifying the resulting equality, to complete the induc-
tion, we must show
(5) 2Sn−1,i =Sn−1,i−1+ 4Sn−2,i +
i−2
X
`=1
Sn−2,`.
Substituting Sn−1,i = 2Sn−2,i +Pi−1
`=1 Sn−2,` into (5) reduces it to
2
i−1
X
`=1
Sn−2,` =Sn−1,i−1+
i−2
X
`=1
Sn−2,`,
i.e.,
Sn−1,i−1= 2Sn−2,i−1+
i−2
X
`=1
Sn−2,`,
which is true by the induction hypothesis. This establishes (3) and completes the proof in
the case of (σ, τ) = (1234,1243). Since the associated an,i can be shown to satisfy the same
recurrence and initial conditions for the other two pattern pairs, one obtains the same result
in these cases as well.
To establish the case (1342,1432), we define a bijection with a previous case.
6 T. MANSOUR AND M. SHATTUCK
Theorem 3. The members of Sn,i(1342,1432) having a prescribed set of left-right minima in
specified positions are in one-to-one correspondence with members of Sn,i(1234,1243) having
the same set of left-right minima in the same positions. In particular, |Sn,i(1342,1432)|=
Sn,i.
Proof. We define a bijection fbetween Sn,i(1342,1432) and Sn,i(1234,1243) with the desired
properties as follows. Let π=x1···xn∈ Sn,i (1342,1432) have left-right minima values
ar> ar−1>··· > a1, where ar=x1and a1= 1. Note that since the patterns in both pairs
start with 1, one may always assume a left-right minima plays the role of the 1. Suppose
π=α1β, where αor βis possibly empty. Then let π1be obtained by reversing the order of
all letters in βwithin π0=π, that is, π1=α1β0, where β0denotes the reversal of β. Note
that π1contains no 1234 or 1243 starting with the actual element 1 since all such occurrences
of 1234 or 1243 within π0have been replaced with comparable occurrences of 1432 or 1342,
respectively.
If r= 1 (i.e., if πstarts with 1), then we let f(π) = π1and we are done, so assume r≥2. In
this case, let Sdenote the subsequence comprising all elements of [a2+ 1, n] occurring to the
right of a2in the permutation π1. Let S∗denote the portion of Sto the right of a1within π1.
We then let π2be the permutation obtained from π1as follows. First remove all letters of π1
corresponding to Sand replace them with blanks. Within these blanks, from left to right, we
then write the elements of S∗followed by the reversal of S−S∗to obtain π2. That is, the
elements between a2and a1in π1that belong to [a2+ 1, n] have their relative order reversed
when they are written within the blanks and now follow (instead of precede) the remaining
elements of S. Observe that π2has no occurrences of 1234 or 1243 starting with a2as the
relative order of all elements belonging to [a2+ 1, n] to the right of a2in π2is the reverse of
the order of these same elements in π. To see this, note that only the elements in S−S∗have
their order reversed in going from π1to π2as the order of S∗in π1is already the reversal of
what it was in π. Further, no occurrence of 1234 or 1243 starting with a1can arise during
the transition from π1to π2since the positions of elements in [2, a2−1] do not change during
this transition.
If r= 2, then let f(π) = π2. Otherwise, consider moving the letters that belong to [a3+ 1, n]
and lie to the right of a3within π2in a comparable manner as before, reversing the order of
only those elements occurring between a3and a2. Continue on for subsequently larger iwhere
in the r-th step, one moves letters in [ar+ 1, n] in obtaining πrfrom πr−1. Let f(π) = πr.
Note that πrindeed belongs to Sn(1234,1243) since as one may verify no occurrence of 1234
or 1243 starting with ajfor some j < i can arise during the i-th transition from πi−1to πi
for any i∈[r]. Since each step of the algorithm described above is seen to preserve both
the positions and values of left-right minima, then so does the mapping f(in particular, the
first letter statistic is preserved by f). This implies that the inverse of fmay be found by
reversing each of the rsteps of the algorithm starting with the last step and proceeding in
reverse order.
We next treat the cases (1324,1423) and (1342,1423) together using a generating tree ap-
proach (see, e.g., [19]). Consider forming π∈ Sn(1324,1423) (or Sn(1342,1423)) by inserting
the element 1 within a member ρ∈ Sn−1(1324,1423) (Sn−1(1342,1423), respectively), ex-
pressed using the letters in [2, n]. By an active site within ρ=ρ1··· ρn−1, we mean a position
in which 1 may be inserted without introducing an occurrence of either 1324 or 1423 (and
likewise for (1342,1423)). Given either pair (σ, τ) of patterns under consideration, let un(i, j)
ON A CONJECTURE OF LIN AND KIM 7
denote the number of members of Sn,i(σ, τ ) having exactly jactive sites. Note that un(i, j)
for n≥2 can assume non-zero values only when 1 ≤i≤nand 3 ≤j≤n+ 1.
The un(i, j) are given recursively as follows.
Lemma 4. If 3≤j≤n, then
(6) un(i, j) = un−1(i−1, j −1) +
n
X
`=j−1
un−1(i−1, `),2≤i≤n,
with un(1, j)=0. If j=n+ 1, then we have
(7) un(i, n + 1) = un−1(i−1, n), i ≥2,
with un(1, n + 1) = 2n−2for n≥2and u1(1,2) = 1.
Proof. We treat the case (1324,1423), with the same recurrences seen to hold for (1342,1423)
by a comparable analysis. Let Un,i,j denote the subset of Sn,i(1324,1423) enumerated by
un(i, j). Note that the (active) sites of π∈ Un,i,j correspond to the rightmost jpossible
positions of πin which to insert a 1 if j≤n. For if the j-th letter xfrom the right within
πwhere j≤nstarts either a 213 or 312 and is the rightmost such letter to do so, then all
positions to the right of xare sites. From this observation, we may conclude that Un,1,j is
empty if 3 ≤j≤n, upon considering separately the cases j=nor j < n. On the other
hand, since 1 starts both of the patterns that are being avoided, we have that Un,1,n+1 is
synonymous with Sn−1(213,312), which has cardinality 2n−2if n≥2. This establishes the
initial conditions when i= 1, so assume henceforth that i≥2.
Let πbe formed from a precursor α∈ Un−1,a,b for some aand b, expressed using [2, n],
by inserting 1 as described. Let (k) denote a precursor having ksites. Then we have the
succession rule (k)→(3)(4) ···(k+ 1)(k+ 1) with root (2), which follows from the argument
used in the proof of [8, Prop. 11] or can be reasoned directly in this case. Note that no
offspring of αif b < n can arise by inserting 1 in the first position, for otherwise a 1324
or 1423 would be introduced, whereas if b=n, an offspring so produced would have n+ 1
sites. Since any offspring π∈ Un,i,j where i≥2 must have precursor starting with i−1
and containing at least j−1 sites, recurrence (6) follows. On the other hand, an offspring
π∈ Un,i,n+1 where i≥2 can only come about by inserting 1 in the final position within
its precursor ρ∈ Un−1,i−1,n, for the other offspring of ρhaving n+ 1 sites comes about by
inserting 1 in the first position. This implies (7) and completes the proof.
Given n≥2, let vn(i;q) = Pn+1
j=3 un(i, j)qj−2for 1 ≤i≤n, with v1(1; q) = 1. Define the
joint distribution polynomial vn(y, q) = Pn
i=1 vn(i;q)yifor n≥2, with v1(y, q) = y.
Then the vn(y, q) are given recursively as follows.
Lemma 5. If n≥2, then
(8) vn(y, q) = (1 −y)(2n−1y−yn)qn−1
2−y+yq(vn−1(y, 1) + (1 −2q)vn−1(y, q))
1−q,
with v1(y, q) = y.
Proof. Note that (8) is seen to hold if n= 2 since v2(y, q) = yq(1 + y), so assume n≥3.
Multiplying both sides of (6) by qj−2, summing over 3 ≤j≤nand adding qn−1times (7)
8 T. MANSOUR AND M. SHATTUCK
gives
vn(i;q) = un−1(i−1, n)qn−1+
n
X
j=3
un−1(i−1, j −1)qj−2+
n
X
j=3
qj−2
n
X
`=j−1
un−1(i−1, `)
=q
n
X
j=3
un−1(i−1, j)qj−2+
n
X
`=2
un−1(i−1, `)
`+1
X
j=3
qj−2−un−1(i−1, n)qn−1
=qvn−1(i−1; q) +
n
X
`=2
un−1(i−1, `)q−q`
1−q−un−1(i−1, n)qn−1
=qvn−1(i−1; q) + q
1−q(vn−1(i−1; 1) −qvn−1(i−1; q)) −un−1(i−1, n)qn−1,
for 2 ≤i≤n, with vn(1; q) = 2n−2qn−1. Multiplying the last recurrence by yi, and summing
over i, yields
vn(y, q) = yqvn−1(y, q) + yq(vn−1(y, 1) −qvn−1(y, q))
1−q+ 2n−2yqn−1−qn−1
n
X
i=2
un−1(i−1, n)yi
=yq(vn−1(y, 1) + (1 −2q)vn−1(y, q))
1−q+ 2n−2yqn−1−qn−1 yn+
n−1
X
i=2
2n−i−1yi!,
where we have used the fact
um(j, m + 1) = |Sm,j (213,312)|=(2m−j−1,if 1 ≤j < m;
1,if j=m,
which follows from elementary considerations. Simplifying and combining the inhomogeneous
terms in the last recurrence formula now gives (8).
Note that (8) holds for both (1324,1423) and (1342,1423) since Lemma 4 applies to either
pair. Let f(x, y;q) = Pn≥1vn(y, q)xn.
Theorem 6. The generating function for the joint distribution of the first letter and number
of active sites statistics on Sn(1324,1423) and Sn(1342,1423) is given by
(9) f(x, y;q) = xy(1 −q)(1 −xq)(1 −2xyq)
(1 −2xq)(1 −xyq)(1 −(xy + 1)q+ 2xyq2)+xyq
1−(xy + 1)q+ 2xyq2A(x, y),
where
A(x, y) = xy(2 −3x−3y+ 3xy) + xy(x+y−xy)(xy +p1−6xy +x2y2)
2(1 −2x−y+xy).
Proof. Multiplying both sides of (8) by xn, and summing over n≥2, yields
f(x, y;q)−xy =y(1 −y)
2−y·2x2q
1−2xq −1−y
2−y·x2y2q
1−xyq +xyq
1−qf(x, y; 1)
+xyq(1 −2q)
1−qf(x, y;q),
which may be rewritten as
1−xyq(1 −2q)
1−qf(x, y;q) = xy +x2yq(1 −y)
(1 −2xq)(1 −xyq)+xyq
1−qf(x, y; 1).(10)
ON A CONJECTURE OF LIN AND KIM 9
To solve (10), we apply the kernel method [6] and let
q=q0=1 + xy −p1−6xy +x2y2
4xy .
This gives
f(x, y; 1) = q0−1
q0
+x(1 −y)(q0−1)
1−2xq0−xyq0+ 2x2yq2
0
=q0−1
q0
+x(1 −y)(q0−1)
1−x+x(xy −y−1)q0
,
where we have used the fact 2xyq2
0= (1 + xy)q0−1. Substituting 1
q0=1+xy+√1−6xy+x2y2
2
into
f(x, y; 1) = 1−1
q0 1 + x(1 −y)
1−x
q0+x(xy −y−1)!,
and simplifying, leads to the formula f(x, y; 1) = A(x, y). Substituting this back into (10)
yields (9).
3. The remaining cases
In this section, we consider the remaining cases (1243,1423), (1243,1342) and (1423,1324).
We adopt a common approach for these three cases and consider the joint distribution of the
first and second letter statistics. For the given pattern pair (σ, τ ) under current consideration,
let an(i, j) denote the number of members of Sn,i(σ, τ) whose second letter is j. Let an(i) =
Pj6=ian(i, j) and an=Pn
i=1 an,i. Note that an=Sn, the n-th Schr¨oder number (see, e.g., [8]).
Clearly, we have
(11) an(i, j) = an−1(j),1≤j < i,
for all pairs of patterns under consideration. If i<j where i≤n−3, then in order to write a
recurrence for an(i, j) in this case, we consider the position of the second ascent and further
subcases based on the size of the difference j−i. If n≤3, then all three pattern pairs have
initial values given by a1(1) = a2(1,2) = a2(2,1) = a3(i, j) = 1 where i, j ∈[3] with i6=j.
3.1. The case (1243,1423).In this subsection, we enumerate the members of Sn(1243,1423)
according to the joint distribution of the first and second letter statistics.
We first prove the following recurrence for an(i, j) when i<j.
Lemma 7. We have
(12) an(i, i +1) = an−1(i, i + 1) +
i−1
X
a=1
i−a−1
X
c=0
i−c
X
b=a+1 i−a−1
can−c−2(a, b),1≤i≤n−2,
with an(n−1, n) = an−2,
an(i, i + 2) = an−1(i, i + 1) + an−1(i, i + 2)
+
i−1
X
a=1
i−a−1
X
c=0
i−c+1
X
b=a+1 i−a−1
can−c−2(a, b),1≤i≤n−3,(13)
with an(n−2, n) = an−2, and
an(i, j) = an−1(i, j −1) +
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−2)
10 T. MANSOUR AND M. SHATTUCK
+ (1 −δj,n)· an−1(i, j ) +
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)!
(14)
for 4≤i+ 3 ≤j≤n.
Proof. Let kdenote the third letter of a member of An,i,j. We will consider various cases
based on kin the proofs of (12)-(14). To show (12), first observe that for members of An,i,i+1
where i≤n−2, one has either k=i+ 2 or k < i, for k > i + 2 would ensure an occurrence
of 1243 which isn’t permissible. In the first case, the third letter is superfluous concerning
avoidance of either pattern and thus may be deleted leading to an−1(i, i + 1) possibilities.
On the other hand, in the latter case, we have that π∈ An,i,i+1 can be decomposed as
π=i(i+ 1)x1···xcadρ, where a∈[i−1], x1, . . . , xc∈[a+ 1, i −1] with x1>· ·· > xc,d>a
and ρdenotes the terminal section of π. Note that d=i+ 2 or d∈[a+ 1, i −1] −{x1, . . . , xc}
since d > i+2 is again not allowed. Further, the letters x1, . . . , xc, i, i+1 may be removed from
πin light of the ascent a, d, leading to a member π0∈ An−c−2,a,b for some bafter reduction of
letters. Note that within π0, the bparameter value can range over [a+1, i −c] since the largest
possible value it may assume is i+ 2 −(c+ 2) = i−c. As there are i−a−1
cways in which to
choose the xi, considering all possible a,cand bimplies that the number of π∈ An,i,i+1 for
which k < i is given by Pi−1
a=1 Pi−a−1
c=0 Pi−c
b=a+1 i−a−1
can−c−2(a, b), which establishes (12).
A similar argument applies to (13) except that now we have k < i or k=i+ 1, i + 3 for
members of An,i,i+2 where i≤n−3. Note that k=i+ 3 leads to an−1(i, i + 2) possibilities,
as kmay be deleted in this case. If k < i, then making use of the previous notation, we have
d∈[a+ 1, i −1] ∪{i+ 1, i + 3}and thus bmay range in [a+ 1, i −c+ 1] in this case. Finally, to
show (14), we consider first the case when j < n within a member of An,i,j where j≥i+ 3.
Here, we would have k∈[i−1]∪{j−1, j + 1}, as k∈[i+ 1, j −2] would lead to an occurrence
of 1423 as witnessed by ijk(k+ 1). If k=j−1 or k=j+ 1, then kmay be deleted in either
case giving an−1(i, j −1) and an−1(i, j) possibilities, respectively. If k < i, then we must have
d=j−1 or d=j+ 1, for other values of dwould lead to an occurrence of 1243 or 1423.
Considering all possible aand cthen yields the two double sum expressions occurring in the
j < n case of (14). Combining this with the preceding then implies (14) when j < n. On the
other hand, if j=n, then k=j+ 1 or d=j+ 1 does not occur in the preceding argument,
which implies (14) in this case and completes the proof.
We will make use of the following generating functions:
A(x, v, w) = X
n≥2
n
X
a=1
n
X
b=1
an(a, b)vawbxn,
A+(x, v, w) = X
n≥2
n−1
X
a=1
n
X
b=a+1
an(a, b)vawbxn,
A−(x, v, w) = X
n≥2
n
X
a=2
a−1
X
b=1
an(a, b)vawbxn,
C(x, v) = X
n≥2
n−1
X
a=1
an(a, a + 1)vaxn,
ON A CONJECTURE OF LIN AND KIM 11
D(x, v) = X
n≥3
n−2
X
a=1
an(a, a + 2)vaxn,
B(x, v, w) = X
n≥4
n−3
X
a=1
n
X
b=a+3
an(a, b)vawbxn.
Translating (11)-(14) in terms of generating functions (the details of which are provided below
in the appendix) yields the following system of functional equations.
Proposition 8. We have
A+(x, v, w) = wC(x, vw) + w2D(x, vw) + B(x, v, w),
A−(x, v, w) = v2wx2+vx
1−vA(x, vw, 1) −v2x
1−vA(vx, w, 1),
C(x, vw) = vwx2A(vwx, 1,1) + vwx2+v2w2x3+xC(x, vw)
+vwx2
vwx +vw −1A+(vwx
1−vwx ,1−vwx, 1)
−(1 −vwx)x2
vwx +vw −1A+(x, 1−vwx, vw
1−vwx ),
D(x, vw) = x2A(v wx, 1,1) −v2w2x4+x(C(x, vw)−vwx2A(vwx, 1,1)) + xD(x, vw)
+x2(1 −vwx)
vwx +vw −1A+(vwx
1−vwx ,1−vwx, 1)
−x2(1 −vwx)2
vw(vwx +vw −1)A+(x, 1−vwx, vw
1−vwx )−x2C(x, vw),
B(x, v, w) = wxB(x, v, w) + w3xD(x, v w)
+w2x2(1 −vwx)
vwx +v−1A+(x, 1−vwx, vw
1−vwx )−vw2x2
vwx +v−1A+(x, v , w)
+xB(x, v, w) + wx2(1 −vwx)2
v(vwx +v−1) B(x, 1−vwx, vw
1−vwx )−vwx2
vwx +v−1B(x, v , w).
Using the conjecture itself as an aid, we are able to solve explicitly (see appendix for details)
the preceding system and obtain the following result.
Theorem 9. The generating function for the joint distribution of the first and second letter
statistics on Sn(1243,1423) for n≥2is given by
A+(x, v, w) + A−(x, v , w),
where A+(x, v, w)and A−(x, v, w)are as in (28) and (29), respectively.
Substituting w= 1 in the prior theorem and finding vx +A(x, v, 1), one obtains the following
result.
Corollary 10. The generating function for the distribution of the first letter statistic on
Sn(1243,1423) for n≥1is given by
vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)
2(1 −v−2x+vx).
12 T. MANSOUR AND M. SHATTUCK
3.2. The case (1243,1342).We first write a recurrence for an(i, j) when i<j.
Lemma 11. If 1≤i<j≤n−1, then
an(i, j) =
j−1
X
k=i+1
an−1(i, k) +
i−1
X
a=1
i−a−1
X
c=0
j−c−2
X
b=a+1 i−a−1
can−c−2(a, b)
+δi+1,j · an−1(i, i + 1) +
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, i −c)!,(15)
with an(i, n) = an−1(i)for 1≤i≤n−1.
Proof. The formula when j=nis obvious, so assume j < n. Let An,i,j denote the subset
of Sn,i(1243,1342) having second letter j. Let π∈ An,i,j where 1 ≤i<j≤n−1 and
kdenote the third letter of π. Suppose k < j, noting that this is indeed a requirement if
j≥i+ 2, for otherwise j < n would imply a 1342 would occur. If i+ 1 ≤k≤j−1, then the
letter jis extraneous and thus may be deleted from πsince all elements of [k+ 1, n]− {j}
must occur in increasing order with all elements of [i+ 1, k] occurring to the left of those in
[k+ 1, n]− {j}. This implies the first summation formula in (15). Otherwise k < i and we
may write π=ijx1··· xcadρ, where 1 ≤a < x1<·· · < xc≤i−1 and d > a. Note that
d<jif j≥i+ 2 in order to avoid a 1342 of the form ijdt for some t∈[i+ 1, j −1].
We argue now that d < j implies that iand jmay be deleted from π. Clearly, the letters i
and jare extraneous concerning the avoidance of 1243 in light of the ascent a, d where a < i.
They are also irrelevant concerning 1342, whence they may be deleted. To see this, note
that if d<i, then members of [d+ 1, n] occurring to the right of dmust form an increasing
subsequence due to apreceding dand thus no 1342 can start with ij. The same conclusion
is reached if i<d<j, for in this case all letters in [i+ 1, d −1] occur to the left of those
in [d+ 1, n]− {j}, with the latter forming an increasing subsequence. Since x1, . . . , xcmay
clearly also be deleted from πas a < xc, one is left with π0∈ An−c−2,a,b for some aand b.
Note that b∈[a+ 1, j −c−2] (after reducing letters) since d∈[a+ 1, j −1] −{x1, . . . , xc, i}.
Considering all possible a,cand bthen yields the triple sum expression in (15) and finishes
the case when j≥i+ 2. On the other hand, if j=i+ 1, then k=j+ 1 is possible without
introducing an occurrence of 1342, in which case kmay be deleted resulting in a member of
An−1,i,i+1. Likewise, d=j+ 1 is also possible in the decomposition of πabove. Combining
these two additional cases then accounts for the second line in formula (15) and completes
the proof.
From (15), we may deduce the following further useful formulas.
Lemma 12. If 1≤i≤n−2, then
(16) an(i, i + 1) = an(i, i + 2)
and
(17) an(i, i + 1) =
i
X
`=1
an−1(`, i + 1).
Proof. Both equalities are easily seen to hold if i=n−2, so assume 1 ≤i≤n−3. Taking
j=i+ 1 and j=i+ 2 in (15), and comparing the results, then completes the proof of (16).
ON A CONJECTURE OF LIN AND KIM 13
For (17), first observe
i−1
X
`=1
an−1(`, i + 1) =
i−1
X
`=1
i
X
k=`+1
an−2(`, k) +
i−1
X
`=1
`−1
X
a=1
`−a−1
X
c=0
i−c−1
X
b=a+1 `−a−1
can−c−3(a, b)
=
i−1
X
`=1
i
X
k=`+1
an−2(`, k) +
i−2
X
a=1
i−a−2
X
c=0
i−c−1
X
b=a+1
an−c−3(a, b)
i−1
X
`=a+c+1 `−a−1
c
=
i−1
X
`=1
i
X
k=`+1
an−2(`, k) +
i−2
X
a=1
i−a−1
X
c=1
i−c
X
b=a+1 i−a−1
can−c−2(a, b).
Then, by (16) and (15) when j=i+ 2, we have
an(i, i + 1) −
i−1
X
`=1
an−1(`, i + 1) = an(i, i + 2) −
i−1
X
`=1
an−1(`, i + 1)
=an−1(i, i + 1) −
i−1
X
`=1
i
X
k=`+1
an−2(`, k)
+ i−1
X
a=1
i−a−1
X
c=0
i−c
X
b=a+1 −
i−2
X
a=1
i−a−1
X
c=1
i−c
X
b=a+1!i−a−1
can−c−2(a, b)
=an−1(i, i + 1) −
i−1
X
`=1
i
X
k=`+1
an−2(`, k) +
i−1
X
a=1
i
X
b=a+1
an−2(a, b) = an−1(i, i + 1),
which completes the proof of (17).
To summarize, we have the following recurrence for an(i, j):
an(i, j) = an−1(j),1≤j < i ≤n,
an(i, n) = an−1(i),1≤i≤n−1,
an(i, i + 1) = Pi
`=1 an−1(`, i + 1),1≤i≤n−2,
an(i, i + 2) = an(i, i + 1),1≤i≤n−2,
an(i, j) = Pj−1
k=i+1 an−1(i, k)
+Pi−1
a=1 Pi−a−1
c=0 Pj−c−2
b=a+1 i−a−1
can−c−2(a, b),
4≤i+ 3 ≤j≤n−1.
(18)
In this case, we state the recurrence formulas satisfied by the corresponding distribution
polynomials which will aid in translating (18) to functional equations as they are not too
lengthy. Define A+
n(v, w) = Pn−1
i=1 Pn
j=i+1 an(i, j)viwj,A−
n(v, w) = Pn
i=2 Pi−1
j=1 an(i, j)viwj
and An(v, w) = Pn
i=1 Pn
j=1 an(i, j)viwjfor n≥2. Clearly, An(v, w) = A+
n(v, w) + A−
n(v, w)
for all n≥2. Further, we define
Cn(v) =
n−2
X
i=1
an(i, i + 1)viand Bn(v, w) =
n−4
X
i=1
n−1
X
j=i+3
an(i, j)viwj.
Then (18) may be rewritten in terms of these distributions as
A−
n(v, w) = v
1−vAn−1(vw, 1) −vn+1
1−vAn−1(w, 1),
14 T. MANSOUR AND M. SHATTUCK
A+
n(v, w) = Bn(v , w) + w2(Cn(vw)−(vw)n−2An−2(1,1)) + wCn(vw) + wnAn−1(1, w),
Cn(v) = 1
vA+
n−1(1, v),
Bn(v, w) = w
1−w(Bn−1(v, w) + w2(Cn−1(vw)−An−3(1,1)(vw)n−3))
−wn
1−w(Bn−1(v, 1) + Cn−1(v)−An−3(1,1)vn−3)
+w3
1−w(Cn−1(vw)−An−3(1,1)vn−3wn−3)−wn
1−w(Cn−1(v)−An−3(1,1)vn−3)
+
n−4
X
i=1
n−1
X
j=i+3
i−1
X
a=1
i−a−1
X
c=0
j−c−2
X
b=a+1 i−1−a
can−c−2(a, b)viwj.
Define A±(x, v, w) = Pn≥2A±
n(v, w)xnand A(x, v , w) = Pn≥2An(v, w)xn. Further, define
B(x, v, w) = Pn≥2Bn(v , w)xnand C(x, v) = Pn≥2Cn(v)xn. Rewriting the preceding recur-
rences in terms of generating functions yields the following result.
Proposition 13. We have
A−(x, v, w) = v2wx2+vx
1−vA(x, vw, 1) −v2x
1−vA(vx, w, 1),
A+(x, v, w) = vw2x2−vw3x3+B(x, v, w) + w2(C(x, vw)−x2A(v wx, 1,1))
+wC(x, vw) + wxA(wx, v, 1),
where
C(x, v) = x
vA+(x, 1, v),
B(x, v, w) = wx(1 −v)
(1 −v−vwx)(1 −w)(B(x, v , w)−B(wx, v, 1))
+2(1 −vw)w3x
(1 −vw −vwx)(1 −w)C(x, vw)−2wx(1 −v)
(1 −v−vwx)(1 −w)C(wx, v)
+x2w2(1 −vwx)2
(1 −v−vwx)(1 −vw −vwx)(B(vwx
1−vwx ,1−vwx, 1) −B(x, 1−vwx, vw
1−vwx ))
+2x2w2(1 −vwx)2
(1 −v−vwx)(1 −vw −vwx)C(vwx
1−vwx ,1−vwx).
By mathematical programming, one may verify the following solution of the foregoing system
of functional equations.
Theorem 14. The generating function for the joint distribution of the first and second letter
statistics on Sn(1243,1342) for n≥2is given by A(x, v, w) = A+(x, v , w) + A−(x, v, w),
where
A+(x, v, w)
=(1 −v+ (v2x−2v2+vx −x)w−vx(x−2)(v−1)w2)vw2x2
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw)) pv2w2x2−6vwx + 1
+(1 −v+ (6v2−4v−x(4v2−2v+ 1))w+xv(x(v2+ 4v−4) −2v2−6v+ 6)w2)vw2x2
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))
ON A CONJECTURE OF LIN AND KIM 15
−(x−2)(v−1)v3w5x4
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))
=vw2x2+w2v(vw +w+ 1)x3+w2v(2v2w2+ 2vw2+ 2vw + 2w2+w+ 1)x4+···
and
A−(x, v, w)
=(x(1 + v−2vx) + x(v2x−v2+vx −1)w−v(x−1)(vx −1)w2)v2wx2
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w)) pv2w2x2−6vwx + 1
+((3x−2)(w−1) + (−w2x2−3w2x−2wx2+ 3w2+ 2wx + 6x2−2w−3x)v)v2wx2
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))
+(w2x+wx2−w2+ 3wx −2x2−3w−2x+ 3 −(x−1)(w−1)vwx)v4w2x3
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))
=v2wx2+ (vw +v+ 1)v2wx3+ 2(v2w2+v2w+v2+vw +v+ 1)v2wx4+·· · .
Moreover, the generating functions counting the members of Sn(1243,1342) starting with an
ascent of size greater than two or of size exactly one and whose second letter is not nin either
case according to the first and second letter statistics are given respectively by
B(x, v, w) = (vwx −2wx2+ 2wx +x−1)w2x2
2(1 −2vw −x(1 −vw))(1 −v−wx(2 −v)) pv2w2x2−6vwx + 1
+(1 −x+ (3vx −4v+ 2x−2)wx −(2x+v−6)vw2x2)w2x2
2(1 −2vw −x(1 −vw))(1 −v−wx(2 −v))
= 2vw4x5+ 2(4vw + 2w+ 1)vw4x6+ 217v2w2+ 10vw2+ 5vw + 4w2
+ 2w+ 1vw4x7+· ··
and
C(x, v) = −vx2(vx2−2vx −3x+ 2 + (x−2)√v2x2−6vx + 1)
2(1 −x−2v+vx)
=vx3+ (2v+ 1)vx4+ (6v2+ 3v+ 1)vx5+ (22v3+ 11v2+ 4v+ 1)vx6+· ·· .
Substituting w= 1 in the prior theorem and finding vx +A(x, v, 1), one obtains the following
result.
Corollary 15. The generating function for the distribution of the first letter statistic on
Sn(1243,1342) for n≥1is given by
vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)
2(1 −v−2x+vx).
3.3. The case (1243,1324).We first write a recurrence for an(i, j) when i<j.
Lemma 16. We have
(19) an(i, i + 1) = an−1(i, i + 1) +
i−1
X
a=1
i−a−1
X
c=0
i−c
X
b=a+1 i−a−1
can−c−2(a, b),1≤i≤n−2,
16 T. MANSOUR AND M. SHATTUCK
and
(20) an(i, j) = an−1(i, j )+
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1),3≤i+2 ≤j≤n−1,
with an(i, n) = an−1(i)for 1≤i≤n−1.
Proof. A similar proof may be given as in the prior two cases. Note that the formula for
an(i, n) is obvious since an nin the second position may clearly be removed. Let kdenote
the third letter of a member of Sn,i(1243,1324). If j=i+ 1 where 1 ≤i≤n−2, then either
k=i+ 2 or k < i, where clearly kmay be deleted in the former case. Assuming the latter, let
a, d denote the second leftmost ascent. Then we must have a+ 1 ≤d≤i+ 2, for otherwise a
1243 would occur. Thus, each letter prior to amay be deleted in this case and considering all
possible a,band c, where band care as before, implies formula (19). If i+ 2 ≤j < n, then
we have k=j+ 1 or k < i, the former leading to an−1(i, j) possibilities. On the other hand,
if k < i and a, d denotes the second ascent, then we must have d=j+ 1. To see this, note
that elements of [d+ 1, n] to the right of dmust form an increasing subsequence and thus
d < j < n would imply an occurrence of 1324 of the form ijxn where x∈[i+1, j−1]. Therefore
d=j+ 1 implies all letters prior to aagain may be deleted, resulting in a permutation that
starts a, j −c−1. Considering all possible aand cthen accounts for the double sum expression
in (20) and completes the proof.
From this, one may deduce the following further useful formula.
Lemma 17. If 1≤i≤n−3and i+ 2 ≤j≤n−1, then
(21) an(i, j) = an−1(1, j −1) + an−1(2, j −1) + · ·· +an−1(i−1, j −1) + an−1(i, j).
Proof. Note that (21) is clearly true on combinatorial grounds if i= 1 since the third letter
of a member of Sn(1243,1324) starting with 1, j where j < n must be j+ 1. So let i≥2 and
j∈[i+ 2, n −1]. Then by (20), we have
(22)
i−1
X
`=1
an−1(`, j −1) =
i−1
X
`=1 an−2(`, j −1) +
`−1
X
a=1
`−a−1
X
c=0 `−a−1
can−c−3(a, j −c−2)!.
If i≥3, then
i−1
X
`=2
`−1
X
a=1
`−a−1
X
c=0 `−a−1
can−c−3(a, j −c−2)
=
i−2
X
a=1
i−a−2
X
c=0
an−c−3(a, j −c−2)
i−1
X
`=a+c+1 `−a−1
c
=
i−2
X
a=1
i−a−2
X
c=0
an−c−3(a, j −c−2)i−a−1
c+ 1
=
i−2
X
c=1
i−c−1
X
a=1 i−a−1
can−c−2(a, j −c−1).
ON A CONJECTURE OF LIN AND KIM 17
Thus, the right-hand side of (22) is given by
i−1
X
`=1
an−2(`, j −1) +
i−2
X
c=1
i−c−1
X
a=1 i−a−1
can−c−2(a, j −c−1)
=
i−2
X
c=0
i−c−1
X
a=1 i−a−1
can−c−2(a, j −c−1)
=
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)
=an(i, j)−an−1(i, j ),
again by (20), which completes the proof.
Summarizing, we have the following recurrence relations satisfied by an(i, j):
an(i, j) = an−1(j),1≤j < i ≤n,
an(i, n) = an−1(i),1≤i≤n−1,
an(i, j) = Pi−1
`=1 an−1(`, j −1) + an−1(i, j ),
1≤i≤n−3 and i+ 2 ≤j≤n−1,
an(i, i + 1) = an−1(i, i + 1) + Pi−1
a=1 Pi
b=a+1 Pi−b
c=0 i−a−1
can−c−2(a, b),
1≤i≤n−2.
(23)
Define An(v, w), A±
n(v, w) and Cn(v) as in the previous subsection, but with Bn(v, w) now
given by Bn(v, w) = Pn−3
i=1 Pn−1
j=i+2 an(i, j)viwj. Translating (23) then yields the following
recurrences:
A−
n(v, w) =
n
X
i=2
i−1
X
j=1
an−1(j)viwj=v
1−vAn−1(vw, 1) −vn+1
1−vAn−1(w, 1),
A+
n(v, w) =
n−2
X
i=1
n−1
X
j=i+1
an(i, j)viwj+
n−1
X
i=1
an(i, j)viwn
=
n−3
X
i=1
n−1
X
j=i+2
an(i, j)viwj+w
n−2
X
i=1
an(i, i + 1)(vw)i+wnAn−1(v, 1)
=Bn(v, w) + wCn(vw) + wnAn−1(v , 1),
with
Bn(v, w) =
n−3
X
i=2
n−1
X
j=i+2
i−1
X
k=1
an−1(i, j −1)viwj+
n−3
X
i=1
n−1
X
j=i+2
an−1(i, j)viwj
=w
n−4
X
k=1
n−1
X
j=k+3
j−2
X
i=k+1
an−1(i, j −1)viwj+
n−4
X
i=1
n−2
X
j=i+2
an−1(i, j)viwj+
n−3
X
i=1
an−2(i)viwn−1
=wv
1−vBn−1(v, w)−w
1−vBn−1(1, vw) + Bn−1(v , w) + wn−1An−2(v, 1)
−wn−1vn−2An−3(1,1)
18 T. MANSOUR AND M. SHATTUCK
and
Cn(v) = Cn−1(v) + vn−2An−3(1,1) +
n−2
X
i=1
i−1
X
a=1
i
X
b=a+1
i−b
X
c=0 i−a−1
can−2−c(a, b)vi
=Cn−1(v) + vn−2An−3(1,1) +
n−3
X
a=1
n−2
X
b=a+1
n−2−b
X
c=0
n−2
X
i=b+ci−a−1
can−2−c(a, b)vi.
Define A(x, v, w), A±(x, v , w) and C(x, v) as in the previous subsection, with B(x, v, w) =
Pn≥2Bn(v, w)xnper the new definition for Bn(v, w). From the preceding recurrences, we
obtain the following system of functional equations.
Proposition 18. We have
A−(x, v, w) = v2wx2+vx
1−vA(x, vw, 1) −v2x
1−vA(vx, w, 1),
A+(x, v, w) = vw2x2+B(x, v, w) + wC(x, vw) + wxA(wx, v, 1),
where
B(x, v, w) = wx
1−v(vB(x, v, w)−B(x, 1, v w)) + xB(x, v, w) + wx2A(wx, v, 1)
−vw2x3A(v wx, 1,1) −v2w3x4,
C(x, v) = vx3(1 + vx)
1−x+vx3
1−xA(vx, 1,1)
+x2
(vx +v−1)(1 −x)(vA+(vx
1−vx ,1−vx, 1) −(1 −vx)A+(x, 1−vx, v
1−vx )).
By programming, one may verify that the solution of the foregoing system is given as follows.
Theorem 19. We have A(x, v, w) = A+(x, v, w) + A−(x, v, w), where
A+(x, v, w)
=(1 −v+ (v2x−2v2+vx −x)w−vx(x−2)(v−1)w2)vw2x2
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw)) pv2w2x2−6vwx + 1
+(1 −v+ (6v2−4v−x(4v2−2v+ 1))w+xv(x(v2+ 4v−4) −2v2−6v+ 6)w2)vw2x2
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))
−(x−2)(v−1)v3w5x4
2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))
=vw2x2+w2v(vw +w+ 1)x3+w2v(2v2w2+ 2vw2+ 2vw + 2w2+w+ 1)x4+···
and
A−(x, v, w)
=(x(1 + v−2vx) + x(v2x−v2+vx −1)w−v(x−1)(vx −1)w2)v2wx2
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w)) pv2w2x2−6vwx + 1
+((3x−2)(w−1) + (−w2x2−3w2x−2wx2+ 3w2+ 2wx + 6x2−2w−3x)v)v2wx2
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))
+(w2x+wx2−w2+ 3wx −2x2−3w−2x+ 3 −(x−1)(w−1)vwx)v4w2x3
2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))
ON A CONJECTURE OF LIN AND KIM 19
=v2wx2+ (vw +v+ 1)v2wx3+ 2(v2w2+v2w+v2+vw +v+ 1)v2wx4+·· · .
Moreover,
B(x, v, w)
=−vw2x2(wx2−2wx −x+ 1)
2(vwx −2vw −x+ 1)(vwx −2wx −v+ 1) pv2w2x2−6vwx + 1
−vw2x2(vw2x3−2vw2x2−vwx2+ 3vwx −3wx2+ 2wx +x−1)
2(vwx −2vw −x+ 1)(vwx −2wx −v+ 1)
=vw3x4+w3v(3vw + 2w+ 1)x5+w3v(11v2w2+ 8vw2+ 4vw + 4w2+ 2w+ 1)x6+···
and
C(x, v) = −vx2(vx2−2vx −3x+ 2 + (x−2)√v2x2−6vx + 1)
2(1 −x−2v+vx).
Taking w= 1 in the prior theorem yields the following result.
Corollary 20. The generating function for the distribution of the first letter statistic on
Sn(1243,1324) for n≥1is given by
vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)
2(1 −v−2x+vx).
4. Appendix
In this section, we provide proofs of the formulas obtained in Proposition 8 and Theorem 9.
We omit the proofs of the comparable steps in the derivations of Theorems 14 and 19.
Proof of Proposition 8:
Clearly, A+(x, v, w) = wC(x, vw) + w2D(x, vw) + B(x, v, w), by the definitions. Translating
(11)-(14) in terms of generating functions yields
A−(x, v, w) = X
n≥2
n
X
a=2
a−1
X
b=1
an−1(b)vawbxn
=X
n≥2
n−1
X
b=1
an−1(b)vb+1 −vn+1
1−vwbxn
=v2wx2+vx
1−vA(x, vw, 1) −v2x
1−vA(vx, w, 1),
wC(x, vw)−vw2x2A(vwx, 1,1) −vw2x2−v2w3x3
=wxC(x, vw) + X
n≥2
n−2
X
i=1
i−1
X
a=1
i−a−1
X
c=0
i−c
X
b=a+1 i−a−1
can−c−2(a, b)viwi+1xn
=wxC(x, vw) + wX
a≥1X
c≥0X
i≥1X
n≥i+c+a+2
i+a
X
b=a+1 i+c−1
can−c−2(a, b)(vw)i+a+cxn
=wxC(x, vw) + wX
a≥1X
i≥1X
n≥a+i
i+a
X
b=a+1
an(a, b)(vw)i+axn+2
(1 −vwx)i
20 T. MANSOUR AND M. SHATTUCK
=wxC(x, vw)
+w
vwx +vw −1X
n≥2
n−1
X
a=1
n
X
b=a+1
an(a, b)( xn+2(vw)n+1
(1 −vwx)n−a−xn+2 (vw)b
(1 −vwx)b−a−1)
=wxC(x, vw)
+vw2x2
vwx +vw −1A+(vwx
1−vwx ,1−vwx, 1) −(1 −vwx)wx2
vwx +vw −1A+(x, 1−vwx, vw
1−vwx ),
w2D(x, vw)−w2x2A(v wx, 1,1) + v2w4x4
=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)
+w2X
n≥3
n−3
X
i=1
i−1
X
a=1
i−1−a
X
c=0
i−c+1
X
b=a+1 i−1−a
can−2−c(a, b)(vw)ixn
=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)
+w2X
a≥1X
c≥0X
i≥1X
n≥i+a+3
i+a+1
X
b=a+1 i+c−1
can−2(a, b)(vw)i+a+cxn+c
=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)
+w2x2X
a≥1X
n≥a+2
n
X
b=a+2
an(a, b)(vw)i+axn
(1 −vwx)i+w2x2X
a≥1X
n≥a+2
n−a−1
X
i=1
an(a, a + 1) (vw)i+axn
(1 −vwx)i
=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)
+w2x2(1 −vwx)
vwx +vw −1A+(vwx
1−vwx ,1−vwx, 1) −wx2(1 −vwx)2
v(vwx +vw −1)A+(x, 1−vwx, vw
1−vwx )
−w2x2C(x, vw),
and
B(x, v, w)
=wxB(x, v, w) + w3xD(x, vw)
+X
n≥3
n−3
X
i=1
n
X
j=i+3
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−2)viwjxn
+X
n≥3
n−3
X
i=1
n
X
j=i+3(1 −δj,n)(an−1(i, j )viwjxn
+
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)viwjxn
=wxB(x, v, w) + w3xD(x, vw)
+X
n≥3
n−3
X
i=1
n
X
j=i+3
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−2)viwjxn
ON A CONJECTURE OF LIN AND KIM 21
+X
n≥3
n−3
X
i=1
n−1
X
j=i+3 an−1(i, j)viwjxn+
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)viwjxn!
=wxB(x, v, w) + w3xD(x, vw)
+X
a≥1X
n≥a+2
n
X
j=a+2
j−a−1
X
i=1
an(a, j)vi+awj+2 xn+2
(1 −vwx)i
+X
n≥3
n−3
X
i=1
n−1
X
j=i+3 an−1(i, j)viwjxn+
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)viwjxn!
=wxB(x, v, w) + w3xD(x, vw)
+w2x2(1 −vwx)
vwx +v−1A+(x, 1−vwx, vw
1−vwx )−vw2x2
vwx +v−1A+(x, v , w)
+X
n≥3
n−3
X
i=1
n−1
X
j=i+3 an−1(i, j)viwjxn+
i−1
X
a=1
i−a−1
X
c=0 i−a−1
can−c−2(a, j −c−1)viwjxn!
=wxB(x, v, w) + w3xD(x, vw)
+w2x2(1 −vwx)
vwx +v−1A+(x, 1−vwx, vw
1−vwx )−vw2x2
vwx +v−1A+(x, v , w)
+xB(x, v, w) + 1
vwx +v−1X
a≥1X
n≥a+3
n
X
j=a+3
an(a, j)( vj−1wj+1 xnn+2
(1 −vwx)j−2−a−va+1 wj+1xnn+2)
=wxB(x, v, w) + w3xD(x, vw)
+w2x2(1 −vwx)
vwx +v−1A+(x, 1−vwx, vw
1−vwx )−vw2x2
vwx +v−1A+(x, v , w)
+xB(x, v, w) + wx2(1 −vwx)2
v(vwx +v−1) B(x, 1−vwx, vw
1−vwx )−vwx2
vwx +v−1B(x, v , w).
Combining the results above gives Proposition 8.
Proof of Theorem 9:
By Proposition 8, one may express Bin terms of the generating functions A, C, D and C, D
in terms of A. Substituting these relations into the last equation from Proposition 8, one
obtains
vx −wx −v−x+ 1
vwx +v−1A+(x, v , w)(24)
=w2x2(vwx −v−1)A+(v wx
1−vwx ,1−vwx, 1)
vwx +vw −1
+vwx2(w2−1)(vwx −1)A+(x, 1−vwx, vw
1−vwx )
(vwx +v−1)(vwx +vw −1)
+w2x2(vwx −v−1)A(v wx, 1,1) + x2vw2(vw2x2−vwx −1).
22 T. MANSOUR AND M. SHATTUCK
Replacing wby w/v in (24), we have
v2x−wx −v2−vx +v
v(wx +v−1) A+(x, v, w/v)
(25)
=w2x2(wx −v−1)A+(wx
1−wx ,1−wx, 1)
v2(wx +w−1) +wx2(w2−v2)(wx −1)A+(x, 1−wx, w
1−wx )
v2(wx +v−1)(wx +w−1)
+w2x2
v2(wx −v−1)A(wx, 1,1) + x2w2
v2(w2x2−vwx −v).
We now seek to determine a formula for A+(x, v, w) using (25). First note that by (2) and
the main result from [8], we have
A(x, 1,1) = 1−3x−√1−6x+x2
2,(26)
which implies A+(x, v, 1) may be determined independently of A−(x, v, 1) since Aoccurs
in (25) only through the A(wx, 1,1) term. Suppose for a moment that A(x, v, 1) is as in
Corollary 10. Then the second equation in Proposition 8 above at w= 1, together with the
fact A(x, v, 1) = A+(x, v, 1) + A−(x, v, 1), gives a linear system of equations in the quantities
A+and A−. This yields
A+(x, v, 1) = vx2(1 + v−vx)
2(vx −v−2x+ 1) p1−6vx +v2x2−vx2(v2x2−v2x−4vx + 3v−1)
2(vx −v−2x+ 1) ,(27)
which we will assume for now to aid in solving (25).
Then taking v=v0=1−x+√1−2(1+2w)x+(1+4w)x2
2(1−x)in (25), and using (26) and (27), implies
A+(x, 1−wx, w
1−wx )
=(1 −x)(wx +w−1)
4(wx −w−2x+ 1)(wx −1) p1−6wx +w2x2p(1 −x)(1 −x−4vwx)
+1−w+ (2w2−w−2)x−(w−1)(2w2+ 4w+ 1)x2+w(2w2−2w−1)x3
4(wx −w−2x+ 1)(wx −1) p1−6wx +w2x2
−(wx +w−1)(wx2−wx −3x+ 1)
4(wx −w−2x+ 1)(wx −1) p(1 −x)(1 −x−4vwx)
+w−1−(3w2−4)x+ (8w3+w2−9w−3)x2
4(wx −w−2x+ 1)(wx −1)
+−w(2w3+ 8w2−9w−4)x3+w2(2w2−2w−1)x4
4(wx −w−2x+ 1)(wx −1) .
Substituting this expression into (24), and using (26) and (27), we obtain
A+(x, v, w)(28)
=(w2−1)(1 −x)vwx2√1−6vwx +v2w2x2p(1 −x)(1 −x−4vwx)
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
+(1 −x−2vwx)vw(1 −x)x2√1−6vwx +v2w2x2
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
ON A CONJECTURE OF LIN AND KIM 23
+(1 −2v2x2+ 4v2x−2v2−x2−2x−2vwx(1 −x))vw3x2√1−6vwx +v2w2x2
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
+(1 −w2)(vwx2−vwx −3x+ 1)vwx2p(1 −x)(1 −x−4vwx)
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
+(3(w2−1)x2+ 4(1 −2w)x−w2+ 4w−1)vwx2
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
−((w2−1)x3+ 8(w2+ 1)x2−(7w2+ 4w+ 11)x+ 4w+ 4)v2w2x2
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)
−2((1 −x)(w2x2+x2+ 3x−3) + vwx(1 −x)2)v3w3x2
4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1) .
One may verify using programming that this expression for A+(x, v, w) indeed satisfies (24)
and (27) and thus is the solution of (24) that is sought. We may now determine A−(x, v, w).
By the second equation in Proposition 8, we have
A−(x, v, 1) = v2x2+vx
1−vA(x, v, 1) −v2x
1−vA(vx, 1,1)
=v2x2+vx
1−v(A+(x, v, 1) + A−(x, v, 1)) −v2x
1−vA(vx, 1,1).
Solving for A−(x, v, 1) in this last equation, and using (26) and (27), yields
A−(x, v, 1) = v2x(x−1)(vx2−vx −3x+ 1 + (x−1)√1−6vx +v2x2)
2(vx −v−2x+ 1) .
Hence, by Proposition 8, we get the following explicit formula for A−(x, v, w):
A−(x, v, w)
(29)
=((1 + v−2vx)x+ (v2x−v2+vx −1)wx −(1 −x)(1 −vx)vw2)v2wx2
2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1) p1−6vwx +v2w2x2
+(6vx2−3(v+ 1)x+ 2 −(2v2x3+ 2v(v+ 1)x2−(3v2+ 2v+ 3)x+ 2v+ 2)w)v2wx2
2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1)
+(v2x3−v2x2+vx3+ 3vx2−3vx −x2−3x+ 3 −(1 −x)(1 −vx)vwx)v3w3x2
2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1) .
Combining the formulas for A+(x, v, w) and A−(x, v, w) yields the formula for A(x, v, w).
References
[1] G. E. Andrews and S. Chern, A proof of Lin’s conjecture on inversion sequences avoiding patterns of
relation triples, J. Combin. Theory Ser. A 179 (2021), Art. 105388.
[2] J. Bonin, L. Shapiro and R. Simion, Some q-analogues of the Schr¨oder numbers arising from combinatorial
statistics on lattice paths, J. Statist. Plann. Inference 34 (1993), 35–55.
[3] W. Cao, E. Y. Jin and Z. Lin, Enumeration of inversion sequences avoiding triples of relations, Discrete
Appl. Math. 260 (2019), 86–97.
[4] S. Corteel, M. Martinez, C. D. Savage and M. Weselcouch, Patterns in inversion sequences I, Discrete
Math. Theor. Comput. Sci. 18 (2016), #2.
[5] S. Gire, Arbres, permutations `a motifs exclus et cartes planaire: quelques probl`emes algorithmiques et
combinatoires, Ph.D. These, University of Bordeaux, 1993.
24 T. MANSOUR AND M. SHATTUCK
[6] Q. Hou and T. Mansour, Kernel method and linear recurrence system, J. Comput. Appl. Math. 216(1)
(2008), 227–242.
[7] S. Kitaev, Patterns in Permutations and Words, Springer-Verlag, Berlin, 2011.
[8] D. Kremer, Permutations with forbidden subsequences and a generalized Schr¨oder number, Discrete Math.
218 (2000), 121–130.
[9] Z. Lin, Restricted inversion sequences and enhanced 3-noncrossing partitions, European J. Combin. 70
(2018), 202–211.
[10] Z. Lin, Patterns of relation triples in inversion and ascent sequences, Theoret. Comput. Sci. 804 (2020),
115–125.
[11] Z. Lin and D. Kim, Refined restricted inversion sequences, arXiv:1706.07208v2, accepted, Ann. Comb.
(2021).
[12] Z. Lin and S. H. F. Yan, Vincular patterns in inversion sequences, Appl. Math. Comput. 364 (2020), Art.
124672.
[13] T. Mansour and M. Shattuck, Pattern avoidance in inversion sequences, Pure Math. Appl. (PU.M.A.) 25
(2015), 157–176.
[14] E. Pergola and R. A. Sulanke, Schr¨oder triangles, paths and parallelogram polyominoes, J. Integer Seq.
1(1998), Art. 98.1.7.
[15] J. L. Ram´ırez and V. F. Sirvent, Generalized Schr¨oder matrix and its combinatorial interpretation, Linear
Multilinear Algebra 66 (2018), 418–433.
[16] D. G. Rogers, A Schr¨oder triangle: Three combinatorial problems, Combinatorial Mathematics V, Lecture
Notes in Mathematics 622, Springer, Berlin, 1977, 175–196.
[17] N. J. Sloane, The On-Line Encyclopedia of Integer Sequences, http://oeis.org, 2010.
[18] J. West, Generating trees and the Catalan and Schr¨oder numbers, Discrete Math. 146 (1995), 247–262.
[19] J. West, Generating trees and forbidden subsequences, Discrete Math. 157 (1996), 363–374.
[20] C. Yan and Z. Lin, Inversion sequences avoiding pairs of patterns, Discrete Math. Theor. Comput. Sci.
22(1) (2020), Art. 23.
Department of Mathematics, University of Haifa, 3498838 Haifa, Israel
E-mail address:tmansour@univ.haifa.ac.il
Department of Mathematics, University of Tennessee, 37996 Knoxville, TN
E-mail address:mshattuc@utk.edu