Content uploaded by Mark Shattuck

Author content

All content in this area was uploaded by Mark Shattuck on Jan 04, 2023

Content may be subject to copyright.

ON A CONJECTURE OF LIN AND KIM CONCERNING A

REFINEMENT OF SCHR ¨

ODER NUMBERS

TOUFIK MANSOUR AND MARK SHATTUCK

Abstract. In this paper, we compute the distribution of the ﬁrst letter statistic on nine

avoidance classes of permutations corresponding to two pairs of patterns of length four. In

particular, we show that the distribution is the same for each class and is given by the entries

of a new Schr¨oder number triangle. This answers in the aﬃrmative a recent conjecture of

Lin and Kim. We employ a variety of techniques to prove our results, including generating

trees, direct bijections and the kernel method. For the latter, we make use of in a creative

way what we are trying to show to aid in solving a system of functional equations satisﬁed

by the associated generating functions in three cases.

1. Introduction

Given two permutations π=π1···πn∈ Snand τ=τ1···τk∈ Sk, we say that πcontains the

pattern τif there exist indices i1< i2<··· < iksuch that πi1···πikis order isomorphic to τ,

that is, πia> πibif and only if τa> τb. Otherwise, πis said to avoid the pattern τ. Moreover,

we say that πavoids a set Lof patterns if it avoids each pattern in L, and let Sn(L) denote

the subset of Snwhose members avoid L. In recent decades, the study of pattern avoidance

in permutations has been the object of considerable attention (see, e.g., [7] and references

contained therein).

An inversion within a permutation σ=σ1···σn∈ Snis an ordered pair (a, b) such that

1≤a<b≤nand σa> σb. The inversion sequence of σis given by a1··· an, where

airecords the number of entries of σto the right of iand less than ifor each i∈[n].

The systematic study of patterns in inversion sequences was initiated only relatively recently

in [4, 13]. Analogous problems, such as avoidance of vincular [12] or multiple [20] patterns,

have been considered on inversion sequences in parallel to those on permutations represented

in the one-line notation.

The (large) Schr¨oder number Sn(see [17, A006318]) is deﬁned recursively by

nSn= 3(2n−3)Sn−1−(n−3)Sn−2, n ≥3,

with S1= 1 and S2= 2, and arises as the enumerator of several avoidance classes of per-

mutations corresponding to a pair of patterns of length four. In particular, combining the

results from [5, 8, 18], one has the Snenumerates Sn(σ, τ) for the following ten inequivalent

pairs (σ, τ):

I. (1234,2134) II. (1324,2314) III. (1342,2341) IV. (3124,3214) V. (3142,3214)

VI. (3412,3421) VII. (1324,2134) VIII. (3124,2314) IX. (2134,3124) X. (2413,3142).

Date: January 3, 2023.

2010 Mathematics Subject Classiﬁcation. 05A15, 05A05.

Key words and phrases. pattern avoidance, combinatorial statistic, kernel method.

1

2 T. MANSOUR AND M. SHATTUCK

This answered in the aﬃrmative a conjecture originally posed by Stanley (see [8] for details).

Moreover, outside of symmetry, there are no other such pairs (σ, τ) for which |Sn(σ, τ)|=Sn.

In this paper, we obtain a reﬁnement of this result in several cases by considering distributions

of certain statistics on the various classes. For other reﬁnements of the Schr¨oder numbers,

see, e.g., [2, 14–16].

n\k123456

1 1

2 1 1

3 2 2 2

4 4 6 6 6

5 8 16 22 22 22

6 16 40 68 90 90 90

Table 1. The new Schr¨oder triangle Sn,k for 1 ≤k≤n≤6.

In a recent paper, Lin and Kim [11] introduced a new triangle Sn,k for Schr¨oder numbers in

their study of inversion sequences; see Table 1 above. Here, we ﬁnd it here more convenient

for what follows to start from k= 1 instead of k= 0, as was done in [11]. Note that Sn,k is

given recursively by

Sn,k =Sn,k−1+ 2Sn−1,k −Sn−1,k−1,1≤k≤n−2,

with Sn,n =Sn,n−1=Sn,n−2for n≥3 and S1,1=S2,1=S2,2= 1. Lin and Kim showed that

Sn,k enumerates the inversion sequences π1···πn∈ {In(021) |πn≡kmod n}. Then they

state the following conjecture which provides a connection between inversion sequences and

pattern avoidance in permutations.

Conjecture 1. (Lin and Kim [11].) Let (ν, µ) be a pair of patterns of length four. Then

Sn,k =|{σ1···σn∈ Sn(ν, µ)|σn=k}|

for all 1 ≤k≤nif and only if (ν, µ) is one of the following nine pairs:

(4321,3421),(3241,2341),(2431,2341),(4231,3241),(4231,2431),

(4231,3421),(2431,3241),(3421,2431),(3421,3241).

Further, it was also shown in [11] that S(n, k) for 1 ≤k≤ngives the cardinality of the

restricted class of inversion sequences {π1···πn∈In(≥,−, >) : πn=k−1}. For other recent

results concerning the avoidance of a pattern of relation triples by inversion sequences, see,

e.g., [1, 3, 9, 10].

Given 1 ≤i≤n, let Sn,i(σ, τ ) denote the set of permutations of length navoiding σand τand

starting with i. Note that one may consider equivalently the distribution of the ﬁrst letter

statistic on the set of permutations avoiding the reversal of the two patterns in question in

each case. Here, we conﬁrm the conjecture by showing that |Sn,i(σ, τ)|=Sn,i for each of the

nine pairs above (where the patterns in each pair are reversed). It is seen that these nine cases

are derived from only six of the ten symmetry classes (I)-(X) above. It should be remarked

that the reversal, complement and inverse operations do not respect the ﬁrst letter statistic

and thus members of the same symmetry class do not have the same ﬁrst letter distribution

in general. Moreover, these are the only nine pairs such that |Sn,i(σ, τ )|=Sn,i for all i; see

Table 2 below which rules out all other possible pairs (σ, τ) for which |Sn(σ, τ)|=Sn.

ON A CONJECTURE OF LIN AND KIM 3

In several cases, we will make use of a generating function approach to establish the result.

Note that by the kernel method [6], one can show

(1)

X

n≥1 n

X

k=1

Sn,kyk!xn=xy(2 −3x−3y+ 3xy) + xy(x+y−xy)(xy +p1−6xy +x2y2)

2(1 −2x−y+xy),

which reduces when y= 1 to the well-known formula for the Schr¨oder number generating

function given by

(2) X

n≥1

Snxn=1−x−√1−6x+x2

2.

σ, τ {|{π∈S8(σ, τ )|π1=k}|}8

k=1 σ, τ {|{π∈S8(σ, τ )|π1=k}|}8

k=1

2413,4123 1584,1036,996,956,879,751,533,233 3142,4123 1584,1584,1252,912,637,443,323,233

3142,3214 1584,1584,736,396,292,304,488,1584 2341,2413 1584,488,304,292,396,736,1584,1584

2341,3142 1584,811,587,489,481,577,855,1584 2413,3214 1584,855,577,481,489,587,811,1584

2431,3421 1806,1022,710,614,644,795,1161,1806 2431,4231 1806,1092,1008,1045,1120,1134,924,429

2314,3124 1806,1092,752,629,629,752,1092,1806 2314,3214 1806,1092,752,629,629,752,1092,1806

2341,3241 1806,1092,752,629,629,752,1092,1806 2413,3142 1806,1092,752,629,629,752,1092,1806

2431,3241 1806,1092,752,629,629,752,1092,1806 2134,3124 1806,1161,795,644,614,710,1022,1806

3241,3421 1806,1806,1198,678,406,342,516,1806 3214,3241 1806,1806,1220,672,390,342,516,1806

3124,4123 1806,1806,1502,1152,840,594,429,429 3214,4213 1806,1806,1502,1152,840,594,429,429

3241,4231 1806,1806,1502,1152,840,594,429,429 3412,4312 1806,1806,1502,1152,840,594,429,429

3421,4231 1806,1806,1502,1152,840,594,429,429 3421,4321 1806,1806,1502,1152,840,594,429,429

4123,4132 1806,1806,1806,1412,928,512,224,64 4123,4213 1806,1806,1806,1412,928,512,224,64

4132,4213 1806,1806,1806,1412,928,512,224,64 4132,4231 1806,1806,1806,1412,928,512,224,64

4132,4312 1806,1806,1806,1412,928,512,224,64 4213,4231 1806,1806,1806,1412,928,512,224,64

4213,4312 1806,1806,1806,1412,928,512,224,64 4231,4312 1806,1806,1806,1412,928,512,224,64

4312,4321 1806,1806,1806,1412,928,512,224,64 3124,3214 1806,1806,788,540,484,540,788,1806

3412,3421 1806,1806,788,540,484,540,788,1806 2314,2341 1806,516,342,390,672,1220,1806,1806

2134,2314 1806,516,342,406,678,1198,1806,1806 2134,2143 1806,788,540,484,540,788,1806,1806

2341,2431 1806,788,540,484,540,788,1806,1806 1432,2413 233,323,443,637,912,1252,1584,1584

1432,3142 233,533,751,879,956,996,1036,1584 1234,2134 429,429,594,840,1152,1502,1806,1806

1243,2143 429,429,594,840,1152,1502,1806,1806 1324,2134 429,429,594,840,1152,1502,1806,1806

1324,2314 429,429,594,840,1152,1502,1806,1806 1342,2341 429,429,594,840,1152,1502,1806,1806

1432,2431 429,429,594,840,1152,1502,1806,1806 1324,3124 429,924,1134,1120,1045,1008,1092,1806

1423,4123 429,924,1344,1582,1582,1344,924,429 1432,4132 429,924,1344,1582,1582,1344,924,429

1234,1243 64,224,512,928,1412,1806,1806,1806 1243,1324 64,224,512,928,1412,1806,1806,1806

1243,1342 64,224,512,928,1412,1806,1806,1806 1243,1423 64,224,512,928,1412,1806,1806,1806

1324,1342 64,224,512,928,1412,1806,1806,1806 1324,1423 64,224,512,928,1412,1806,1806,1806

1342,1423 64,224,512,928,1412,1806,1806,1806 1342,1432 64,224,512,928,1412,1806,1806,1806

1423,1432 64,224,512,928,1412,1806,1806,1806

Table 2. All symmetries of classes I-X according to ﬁrst letter statistic. Note

the 9 boldface cases corresponding to the conjecture and the 9 italicized cases

obtained by complementation.

This paper is organized as follows. In the next section, we show six cases of the conjecture

above using various methods such as induction, bijections and generating trees. In the third

section, we prove the remaining three cases, each of which involves 1243 and another pattern,

4 T. MANSOUR AND M. SHATTUCK

by considering the joint distribution of the ﬁrst and second letter statistics. This permits one

to write a system of recurrence relations in each case which may then be expressed in terms

of some auxiliary generating functions leading to a system of functional equations. At this

point, one can use the conjecture itself in these particular cases along with the kernel method

to ascertain a potential solution to the aforementioned system, which may then be shown to

be the actual solution. Taking the variable that marks the second letter statistic to be unity

then recovers formula (1) and demonstrates the desired equality of distributions.

We remark further that in order to prove the ﬁnal three cases we make use of a special

decomposition of the respective generating functions into what we term positive and negative

parts as well as parts corresponding to certain ordered pair values of the parameters in

question (see, for example, the various generating functions that are deﬁned following the

proof of Lemma 7 below). Such a decomposition is necessary in translating the system of

recurrences at hand into a system of functional equations. In addition, the decompositions

of the permutations themselves required in deriving the recurrences for the last three cases

involving 1243 (see proofs of Lemmas 7, 11 and 16) are apparently new. Finally, in establishing

the recurrences in the cases (1324,1423) and (1342,1423) (see Lemma 4), we make use of

generating trees in not only tracking the labels of oﬀspring but also the value of the ﬁrst

letter statistic in the oﬀspring. We do not know of other examples of generating trees being

used in this way in conjunction with a statistic auxiliary to the tree structure such as the one

recording the ﬁrst letter.

In several instances, the distribution of the ﬁrst letter statistic on the pattern pair in question

follows as a special case of a more general distribution. For one’s reference, listed below are

the places within the paper where the speciﬁc cases are shown.

Pattern Pair Reference Pattern Pair Reference Pattern Pair Reference

1234,1243 Theorem 2 1243,1324 Corollary 20 1243,1342 Corollary 15

1243,1423 Corollary 10 1324,1342 Theorem 2 1324,1423 Theorem 6

1342,1423 Theorem 6 1342,1432 Theorem 3 1423,1432 Theorem 2

Table 3. Places where speciﬁc cases of Conjecture 1 are proven.

2. Pattern avoidance and the new Schr¨

oder triangle

In this section, we enumerate members of Sn,i(σ, τ ) conﬁrming the conjecture in six of the

cases, the ﬁrst three of which we treat together in the following result.

Theorem 2. If n≥1and 1≤i≤n, then |Sn,i(σ, τ)|=Sn,i for (σ, τ) = (1234,1243),

(1324,1342) and (1423,1432).

Proof. We prove the case when (σ, τ) = (1234,1243), the others being similar. Let An,i =

Sn,i(1234,1243) and we ﬁrst write a recurrence for an,i =|An,i|. Note that members of An,i

where 1 ≤i≤n−3 must have second letter `∈[i−1] or `=n−1, n, for otherwise there

would be an occurrence of 1234 or 1243 starting with i`. Furthermore, the ﬁrst letter iis

seen to be extraneous concerning the avoidance of 1234 or 1243 if `∈[i−1] and thus may be

deleted. Similarly, `may be deleted in cases when `=n−1 or n. This implies the recurrence

an,i = 2an−1,i +

i−1

X

`=1

an−1,`,1≤i≤n−2,

ON A CONJECTURE OF LIN AND KIM 5

with an,n =an,n−1=Pn−1

i=1 an−1,i for n≥3.

Before proceeding further, note that

Sn,n−2=

n−2

X

k=1

(Sn,k −Sn,k−1) =

n−2

X

k=1

(2Sn−1,k −Sn−1,k−1)=2Sn−1,n−2+

n−3

X

k=1

Sn−1,k

=

n−1

X

k=1

Sn−1,k.

Thus, the an,i and Sn,i are both given by the sum of the entries of the previous row if

i∈[n−2, n], with the two arrays also agreeing for n= 1,2. Therefore, to complete the proof,

it suﬃces to show

(3) Sn,i = 2Sn−1,i +

i−1

X

`=1

Sn−1,`,1≤i≤n−2.

To do so, we proceed by induction on nand i, the i= 1 case clearly holding since Sn,1=

2Sn−1,1for all n≥3. So assume n≥4 and 2 ≤i≤n−2. Note further that (3) also

holds when i=n−1 for n≥3, which follows from the work above. Then by the induction

hypothesis, we have

Sn,i =Sn,i−1+ 2Sn−1,i −Sn−1,i−1= 2Sn−1,i−1+

i−2

X

`=1

Sn−1,` + 4Sn−2,i + 2

i−1

X

`=1

Sn−2,`

−2Sn−2,i−1−

i−2

X

`=1

Sn−2,`

= 2Sn−1,i−1+ 4Sn−2,i +

i−2

X

`=1

(Sn−1,` +Sn−2,`).(4)

Upon substituting (3) into (4), and simplifying the resulting equality, to complete the induc-

tion, we must show

(5) 2Sn−1,i =Sn−1,i−1+ 4Sn−2,i +

i−2

X

`=1

Sn−2,`.

Substituting Sn−1,i = 2Sn−2,i +Pi−1

`=1 Sn−2,` into (5) reduces it to

2

i−1

X

`=1

Sn−2,` =Sn−1,i−1+

i−2

X

`=1

Sn−2,`,

i.e.,

Sn−1,i−1= 2Sn−2,i−1+

i−2

X

`=1

Sn−2,`,

which is true by the induction hypothesis. This establishes (3) and completes the proof in

the case of (σ, τ) = (1234,1243). Since the associated an,i can be shown to satisfy the same

recurrence and initial conditions for the other two pattern pairs, one obtains the same result

in these cases as well.

To establish the case (1342,1432), we deﬁne a bijection with a previous case.

6 T. MANSOUR AND M. SHATTUCK

Theorem 3. The members of Sn,i(1342,1432) having a prescribed set of left-right minima in

speciﬁed positions are in one-to-one correspondence with members of Sn,i(1234,1243) having

the same set of left-right minima in the same positions. In particular, |Sn,i(1342,1432)|=

Sn,i.

Proof. We deﬁne a bijection fbetween Sn,i(1342,1432) and Sn,i(1234,1243) with the desired

properties as follows. Let π=x1···xn∈ Sn,i (1342,1432) have left-right minima values

ar> ar−1>··· > a1, where ar=x1and a1= 1. Note that since the patterns in both pairs

start with 1, one may always assume a left-right minima plays the role of the 1. Suppose

π=α1β, where αor βis possibly empty. Then let π1be obtained by reversing the order of

all letters in βwithin π0=π, that is, π1=α1β0, where β0denotes the reversal of β. Note

that π1contains no 1234 or 1243 starting with the actual element 1 since all such occurrences

of 1234 or 1243 within π0have been replaced with comparable occurrences of 1432 or 1342,

respectively.

If r= 1 (i.e., if πstarts with 1), then we let f(π) = π1and we are done, so assume r≥2. In

this case, let Sdenote the subsequence comprising all elements of [a2+ 1, n] occurring to the

right of a2in the permutation π1. Let S∗denote the portion of Sto the right of a1within π1.

We then let π2be the permutation obtained from π1as follows. First remove all letters of π1

corresponding to Sand replace them with blanks. Within these blanks, from left to right, we

then write the elements of S∗followed by the reversal of S−S∗to obtain π2. That is, the

elements between a2and a1in π1that belong to [a2+ 1, n] have their relative order reversed

when they are written within the blanks and now follow (instead of precede) the remaining

elements of S. Observe that π2has no occurrences of 1234 or 1243 starting with a2as the

relative order of all elements belonging to [a2+ 1, n] to the right of a2in π2is the reverse of

the order of these same elements in π. To see this, note that only the elements in S−S∗have

their order reversed in going from π1to π2as the order of S∗in π1is already the reversal of

what it was in π. Further, no occurrence of 1234 or 1243 starting with a1can arise during

the transition from π1to π2since the positions of elements in [2, a2−1] do not change during

this transition.

If r= 2, then let f(π) = π2. Otherwise, consider moving the letters that belong to [a3+ 1, n]

and lie to the right of a3within π2in a comparable manner as before, reversing the order of

only those elements occurring between a3and a2. Continue on for subsequently larger iwhere

in the r-th step, one moves letters in [ar+ 1, n] in obtaining πrfrom πr−1. Let f(π) = πr.

Note that πrindeed belongs to Sn(1234,1243) since as one may verify no occurrence of 1234

or 1243 starting with ajfor some j < i can arise during the i-th transition from πi−1to πi

for any i∈[r]. Since each step of the algorithm described above is seen to preserve both

the positions and values of left-right minima, then so does the mapping f(in particular, the

ﬁrst letter statistic is preserved by f). This implies that the inverse of fmay be found by

reversing each of the rsteps of the algorithm starting with the last step and proceeding in

reverse order.

We next treat the cases (1324,1423) and (1342,1423) together using a generating tree ap-

proach (see, e.g., [19]). Consider forming π∈ Sn(1324,1423) (or Sn(1342,1423)) by inserting

the element 1 within a member ρ∈ Sn−1(1324,1423) (Sn−1(1342,1423), respectively), ex-

pressed using the letters in [2, n]. By an active site within ρ=ρ1··· ρn−1, we mean a position

in which 1 may be inserted without introducing an occurrence of either 1324 or 1423 (and

likewise for (1342,1423)). Given either pair (σ, τ) of patterns under consideration, let un(i, j)

ON A CONJECTURE OF LIN AND KIM 7

denote the number of members of Sn,i(σ, τ ) having exactly jactive sites. Note that un(i, j)

for n≥2 can assume non-zero values only when 1 ≤i≤nand 3 ≤j≤n+ 1.

The un(i, j) are given recursively as follows.

Lemma 4. If 3≤j≤n, then

(6) un(i, j) = un−1(i−1, j −1) +

n

X

`=j−1

un−1(i−1, `),2≤i≤n,

with un(1, j)=0. If j=n+ 1, then we have

(7) un(i, n + 1) = un−1(i−1, n), i ≥2,

with un(1, n + 1) = 2n−2for n≥2and u1(1,2) = 1.

Proof. We treat the case (1324,1423), with the same recurrences seen to hold for (1342,1423)

by a comparable analysis. Let Un,i,j denote the subset of Sn,i(1324,1423) enumerated by

un(i, j). Note that the (active) sites of π∈ Un,i,j correspond to the rightmost jpossible

positions of πin which to insert a 1 if j≤n. For if the j-th letter xfrom the right within

πwhere j≤nstarts either a 213 or 312 and is the rightmost such letter to do so, then all

positions to the right of xare sites. From this observation, we may conclude that Un,1,j is

empty if 3 ≤j≤n, upon considering separately the cases j=nor j < n. On the other

hand, since 1 starts both of the patterns that are being avoided, we have that Un,1,n+1 is

synonymous with Sn−1(213,312), which has cardinality 2n−2if n≥2. This establishes the

initial conditions when i= 1, so assume henceforth that i≥2.

Let πbe formed from a precursor α∈ Un−1,a,b for some aand b, expressed using [2, n],

by inserting 1 as described. Let (k) denote a precursor having ksites. Then we have the

succession rule (k)→(3)(4) ···(k+ 1)(k+ 1) with root (2), which follows from the argument

used in the proof of [8, Prop. 11] or can be reasoned directly in this case. Note that no

oﬀspring of αif b < n can arise by inserting 1 in the ﬁrst position, for otherwise a 1324

or 1423 would be introduced, whereas if b=n, an oﬀspring so produced would have n+ 1

sites. Since any oﬀspring π∈ Un,i,j where i≥2 must have precursor starting with i−1

and containing at least j−1 sites, recurrence (6) follows. On the other hand, an oﬀspring

π∈ Un,i,n+1 where i≥2 can only come about by inserting 1 in the ﬁnal position within

its precursor ρ∈ Un−1,i−1,n, for the other oﬀspring of ρhaving n+ 1 sites comes about by

inserting 1 in the ﬁrst position. This implies (7) and completes the proof.

Given n≥2, let vn(i;q) = Pn+1

j=3 un(i, j)qj−2for 1 ≤i≤n, with v1(1; q) = 1. Deﬁne the

joint distribution polynomial vn(y, q) = Pn

i=1 vn(i;q)yifor n≥2, with v1(y, q) = y.

Then the vn(y, q) are given recursively as follows.

Lemma 5. If n≥2, then

(8) vn(y, q) = (1 −y)(2n−1y−yn)qn−1

2−y+yq(vn−1(y, 1) + (1 −2q)vn−1(y, q))

1−q,

with v1(y, q) = y.

Proof. Note that (8) is seen to hold if n= 2 since v2(y, q) = yq(1 + y), so assume n≥3.

Multiplying both sides of (6) by qj−2, summing over 3 ≤j≤nand adding qn−1times (7)

8 T. MANSOUR AND M. SHATTUCK

gives

vn(i;q) = un−1(i−1, n)qn−1+

n

X

j=3

un−1(i−1, j −1)qj−2+

n

X

j=3

qj−2

n

X

`=j−1

un−1(i−1, `)

=q

n

X

j=3

un−1(i−1, j)qj−2+

n

X

`=2

un−1(i−1, `)

`+1

X

j=3

qj−2−un−1(i−1, n)qn−1

=qvn−1(i−1; q) +

n

X

`=2

un−1(i−1, `)q−q`

1−q−un−1(i−1, n)qn−1

=qvn−1(i−1; q) + q

1−q(vn−1(i−1; 1) −qvn−1(i−1; q)) −un−1(i−1, n)qn−1,

for 2 ≤i≤n, with vn(1; q) = 2n−2qn−1. Multiplying the last recurrence by yi, and summing

over i, yields

vn(y, q) = yqvn−1(y, q) + yq(vn−1(y, 1) −qvn−1(y, q))

1−q+ 2n−2yqn−1−qn−1

n

X

i=2

un−1(i−1, n)yi

=yq(vn−1(y, 1) + (1 −2q)vn−1(y, q))

1−q+ 2n−2yqn−1−qn−1 yn+

n−1

X

i=2

2n−i−1yi!,

where we have used the fact

um(j, m + 1) = |Sm,j (213,312)|=(2m−j−1,if 1 ≤j < m;

1,if j=m,

which follows from elementary considerations. Simplifying and combining the inhomogeneous

terms in the last recurrence formula now gives (8).

Note that (8) holds for both (1324,1423) and (1342,1423) since Lemma 4 applies to either

pair. Let f(x, y;q) = Pn≥1vn(y, q)xn.

Theorem 6. The generating function for the joint distribution of the ﬁrst letter and number

of active sites statistics on Sn(1324,1423) and Sn(1342,1423) is given by

(9) f(x, y;q) = xy(1 −q)(1 −xq)(1 −2xyq)

(1 −2xq)(1 −xyq)(1 −(xy + 1)q+ 2xyq2)+xyq

1−(xy + 1)q+ 2xyq2A(x, y),

where

A(x, y) = xy(2 −3x−3y+ 3xy) + xy(x+y−xy)(xy +p1−6xy +x2y2)

2(1 −2x−y+xy).

Proof. Multiplying both sides of (8) by xn, and summing over n≥2, yields

f(x, y;q)−xy =y(1 −y)

2−y·2x2q

1−2xq −1−y

2−y·x2y2q

1−xyq +xyq

1−qf(x, y; 1)

+xyq(1 −2q)

1−qf(x, y;q),

which may be rewritten as

1−xyq(1 −2q)

1−qf(x, y;q) = xy +x2yq(1 −y)

(1 −2xq)(1 −xyq)+xyq

1−qf(x, y; 1).(10)

ON A CONJECTURE OF LIN AND KIM 9

To solve (10), we apply the kernel method [6] and let

q=q0=1 + xy −p1−6xy +x2y2

4xy .

This gives

f(x, y; 1) = q0−1

q0

+x(1 −y)(q0−1)

1−2xq0−xyq0+ 2x2yq2

0

=q0−1

q0

+x(1 −y)(q0−1)

1−x+x(xy −y−1)q0

,

where we have used the fact 2xyq2

0= (1 + xy)q0−1. Substituting 1

q0=1+xy+√1−6xy+x2y2

2

into

f(x, y; 1) = 1−1

q0 1 + x(1 −y)

1−x

q0+x(xy −y−1)!,

and simplifying, leads to the formula f(x, y; 1) = A(x, y). Substituting this back into (10)

yields (9).

3. The remaining cases

In this section, we consider the remaining cases (1243,1423), (1243,1342) and (1423,1324).

We adopt a common approach for these three cases and consider the joint distribution of the

ﬁrst and second letter statistics. For the given pattern pair (σ, τ ) under current consideration,

let an(i, j) denote the number of members of Sn,i(σ, τ) whose second letter is j. Let an(i) =

Pj6=ian(i, j) and an=Pn

i=1 an,i. Note that an=Sn, the n-th Schr¨oder number (see, e.g., [8]).

Clearly, we have

(11) an(i, j) = an−1(j),1≤j < i,

for all pairs of patterns under consideration. If i<j where i≤n−3, then in order to write a

recurrence for an(i, j) in this case, we consider the position of the second ascent and further

subcases based on the size of the diﬀerence j−i. If n≤3, then all three pattern pairs have

initial values given by a1(1) = a2(1,2) = a2(2,1) = a3(i, j) = 1 where i, j ∈[3] with i6=j.

3.1. The case (1243,1423).In this subsection, we enumerate the members of Sn(1243,1423)

according to the joint distribution of the ﬁrst and second letter statistics.

We ﬁrst prove the following recurrence for an(i, j) when i<j.

Lemma 7. We have

(12) an(i, i +1) = an−1(i, i + 1) +

i−1

X

a=1

i−a−1

X

c=0

i−c

X

b=a+1 i−a−1

can−c−2(a, b),1≤i≤n−2,

with an(n−1, n) = an−2,

an(i, i + 2) = an−1(i, i + 1) + an−1(i, i + 2)

+

i−1

X

a=1

i−a−1

X

c=0

i−c+1

X

b=a+1 i−a−1

can−c−2(a, b),1≤i≤n−3,(13)

with an(n−2, n) = an−2, and

an(i, j) = an−1(i, j −1) +

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−2)

10 T. MANSOUR AND M. SHATTUCK

+ (1 −δj,n)· an−1(i, j ) +

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)!

(14)

for 4≤i+ 3 ≤j≤n.

Proof. Let kdenote the third letter of a member of An,i,j. We will consider various cases

based on kin the proofs of (12)-(14). To show (12), ﬁrst observe that for members of An,i,i+1

where i≤n−2, one has either k=i+ 2 or k < i, for k > i + 2 would ensure an occurrence

of 1243 which isn’t permissible. In the ﬁrst case, the third letter is superﬂuous concerning

avoidance of either pattern and thus may be deleted leading to an−1(i, i + 1) possibilities.

On the other hand, in the latter case, we have that π∈ An,i,i+1 can be decomposed as

π=i(i+ 1)x1···xcadρ, where a∈[i−1], x1, . . . , xc∈[a+ 1, i −1] with x1>· ·· > xc,d>a

and ρdenotes the terminal section of π. Note that d=i+ 2 or d∈[a+ 1, i −1] −{x1, . . . , xc}

since d > i+2 is again not allowed. Further, the letters x1, . . . , xc, i, i+1 may be removed from

πin light of the ascent a, d, leading to a member π0∈ An−c−2,a,b for some bafter reduction of

letters. Note that within π0, the bparameter value can range over [a+1, i −c] since the largest

possible value it may assume is i+ 2 −(c+ 2) = i−c. As there are i−a−1

cways in which to

choose the xi, considering all possible a,cand bimplies that the number of π∈ An,i,i+1 for

which k < i is given by Pi−1

a=1 Pi−a−1

c=0 Pi−c

b=a+1 i−a−1

can−c−2(a, b), which establishes (12).

A similar argument applies to (13) except that now we have k < i or k=i+ 1, i + 3 for

members of An,i,i+2 where i≤n−3. Note that k=i+ 3 leads to an−1(i, i + 2) possibilities,

as kmay be deleted in this case. If k < i, then making use of the previous notation, we have

d∈[a+ 1, i −1] ∪{i+ 1, i + 3}and thus bmay range in [a+ 1, i −c+ 1] in this case. Finally, to

show (14), we consider ﬁrst the case when j < n within a member of An,i,j where j≥i+ 3.

Here, we would have k∈[i−1]∪{j−1, j + 1}, as k∈[i+ 1, j −2] would lead to an occurrence

of 1423 as witnessed by ijk(k+ 1). If k=j−1 or k=j+ 1, then kmay be deleted in either

case giving an−1(i, j −1) and an−1(i, j) possibilities, respectively. If k < i, then we must have

d=j−1 or d=j+ 1, for other values of dwould lead to an occurrence of 1243 or 1423.

Considering all possible aand cthen yields the two double sum expressions occurring in the

j < n case of (14). Combining this with the preceding then implies (14) when j < n. On the

other hand, if j=n, then k=j+ 1 or d=j+ 1 does not occur in the preceding argument,

which implies (14) in this case and completes the proof.

We will make use of the following generating functions:

A(x, v, w) = X

n≥2

n

X

a=1

n

X

b=1

an(a, b)vawbxn,

A+(x, v, w) = X

n≥2

n−1

X

a=1

n

X

b=a+1

an(a, b)vawbxn,

A−(x, v, w) = X

n≥2

n

X

a=2

a−1

X

b=1

an(a, b)vawbxn,

C(x, v) = X

n≥2

n−1

X

a=1

an(a, a + 1)vaxn,

ON A CONJECTURE OF LIN AND KIM 11

D(x, v) = X

n≥3

n−2

X

a=1

an(a, a + 2)vaxn,

B(x, v, w) = X

n≥4

n−3

X

a=1

n

X

b=a+3

an(a, b)vawbxn.

Translating (11)-(14) in terms of generating functions (the details of which are provided below

in the appendix) yields the following system of functional equations.

Proposition 8. We have

A+(x, v, w) = wC(x, vw) + w2D(x, vw) + B(x, v, w),

A−(x, v, w) = v2wx2+vx

1−vA(x, vw, 1) −v2x

1−vA(vx, w, 1),

C(x, vw) = vwx2A(vwx, 1,1) + vwx2+v2w2x3+xC(x, vw)

+vwx2

vwx +vw −1A+(vwx

1−vwx ,1−vwx, 1)

−(1 −vwx)x2

vwx +vw −1A+(x, 1−vwx, vw

1−vwx ),

D(x, vw) = x2A(v wx, 1,1) −v2w2x4+x(C(x, vw)−vwx2A(vwx, 1,1)) + xD(x, vw)

+x2(1 −vwx)

vwx +vw −1A+(vwx

1−vwx ,1−vwx, 1)

−x2(1 −vwx)2

vw(vwx +vw −1)A+(x, 1−vwx, vw

1−vwx )−x2C(x, vw),

B(x, v, w) = wxB(x, v, w) + w3xD(x, v w)

+w2x2(1 −vwx)

vwx +v−1A+(x, 1−vwx, vw

1−vwx )−vw2x2

vwx +v−1A+(x, v , w)

+xB(x, v, w) + wx2(1 −vwx)2

v(vwx +v−1) B(x, 1−vwx, vw

1−vwx )−vwx2

vwx +v−1B(x, v , w).

Using the conjecture itself as an aid, we are able to solve explicitly (see appendix for details)

the preceding system and obtain the following result.

Theorem 9. The generating function for the joint distribution of the ﬁrst and second letter

statistics on Sn(1243,1423) for n≥2is given by

A+(x, v, w) + A−(x, v , w),

where A+(x, v, w)and A−(x, v, w)are as in (28) and (29), respectively.

Substituting w= 1 in the prior theorem and ﬁnding vx +A(x, v, 1), one obtains the following

result.

Corollary 10. The generating function for the distribution of the ﬁrst letter statistic on

Sn(1243,1423) for n≥1is given by

vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)

2(1 −v−2x+vx).

12 T. MANSOUR AND M. SHATTUCK

3.2. The case (1243,1342).We ﬁrst write a recurrence for an(i, j) when i<j.

Lemma 11. If 1≤i<j≤n−1, then

an(i, j) =

j−1

X

k=i+1

an−1(i, k) +

i−1

X

a=1

i−a−1

X

c=0

j−c−2

X

b=a+1 i−a−1

can−c−2(a, b)

+δi+1,j · an−1(i, i + 1) +

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, i −c)!,(15)

with an(i, n) = an−1(i)for 1≤i≤n−1.

Proof. The formula when j=nis obvious, so assume j < n. Let An,i,j denote the subset

of Sn,i(1243,1342) having second letter j. Let π∈ An,i,j where 1 ≤i<j≤n−1 and

kdenote the third letter of π. Suppose k < j, noting that this is indeed a requirement if

j≥i+ 2, for otherwise j < n would imply a 1342 would occur. If i+ 1 ≤k≤j−1, then the

letter jis extraneous and thus may be deleted from πsince all elements of [k+ 1, n]− {j}

must occur in increasing order with all elements of [i+ 1, k] occurring to the left of those in

[k+ 1, n]− {j}. This implies the ﬁrst summation formula in (15). Otherwise k < i and we

may write π=ijx1··· xcadρ, where 1 ≤a < x1<·· · < xc≤i−1 and d > a. Note that

d<jif j≥i+ 2 in order to avoid a 1342 of the form ijdt for some t∈[i+ 1, j −1].

We argue now that d < j implies that iand jmay be deleted from π. Clearly, the letters i

and jare extraneous concerning the avoidance of 1243 in light of the ascent a, d where a < i.

They are also irrelevant concerning 1342, whence they may be deleted. To see this, note

that if d<i, then members of [d+ 1, n] occurring to the right of dmust form an increasing

subsequence due to apreceding dand thus no 1342 can start with ij. The same conclusion

is reached if i<d<j, for in this case all letters in [i+ 1, d −1] occur to the left of those

in [d+ 1, n]− {j}, with the latter forming an increasing subsequence. Since x1, . . . , xcmay

clearly also be deleted from πas a < xc, one is left with π0∈ An−c−2,a,b for some aand b.

Note that b∈[a+ 1, j −c−2] (after reducing letters) since d∈[a+ 1, j −1] −{x1, . . . , xc, i}.

Considering all possible a,cand bthen yields the triple sum expression in (15) and ﬁnishes

the case when j≥i+ 2. On the other hand, if j=i+ 1, then k=j+ 1 is possible without

introducing an occurrence of 1342, in which case kmay be deleted resulting in a member of

An−1,i,i+1. Likewise, d=j+ 1 is also possible in the decomposition of πabove. Combining

these two additional cases then accounts for the second line in formula (15) and completes

the proof.

From (15), we may deduce the following further useful formulas.

Lemma 12. If 1≤i≤n−2, then

(16) an(i, i + 1) = an(i, i + 2)

and

(17) an(i, i + 1) =

i

X

`=1

an−1(`, i + 1).

Proof. Both equalities are easily seen to hold if i=n−2, so assume 1 ≤i≤n−3. Taking

j=i+ 1 and j=i+ 2 in (15), and comparing the results, then completes the proof of (16).

ON A CONJECTURE OF LIN AND KIM 13

For (17), ﬁrst observe

i−1

X

`=1

an−1(`, i + 1) =

i−1

X

`=1

i

X

k=`+1

an−2(`, k) +

i−1

X

`=1

`−1

X

a=1

`−a−1

X

c=0

i−c−1

X

b=a+1 `−a−1

can−c−3(a, b)

=

i−1

X

`=1

i

X

k=`+1

an−2(`, k) +

i−2

X

a=1

i−a−2

X

c=0

i−c−1

X

b=a+1

an−c−3(a, b)

i−1

X

`=a+c+1 `−a−1

c

=

i−1

X

`=1

i

X

k=`+1

an−2(`, k) +

i−2

X

a=1

i−a−1

X

c=1

i−c

X

b=a+1 i−a−1

can−c−2(a, b).

Then, by (16) and (15) when j=i+ 2, we have

an(i, i + 1) −

i−1

X

`=1

an−1(`, i + 1) = an(i, i + 2) −

i−1

X

`=1

an−1(`, i + 1)

=an−1(i, i + 1) −

i−1

X

`=1

i

X

k=`+1

an−2(`, k)

+ i−1

X

a=1

i−a−1

X

c=0

i−c

X

b=a+1 −

i−2

X

a=1

i−a−1

X

c=1

i−c

X

b=a+1!i−a−1

can−c−2(a, b)

=an−1(i, i + 1) −

i−1

X

`=1

i

X

k=`+1

an−2(`, k) +

i−1

X

a=1

i

X

b=a+1

an−2(a, b) = an−1(i, i + 1),

which completes the proof of (17).

To summarize, we have the following recurrence for an(i, j):

an(i, j) = an−1(j),1≤j < i ≤n,

an(i, n) = an−1(i),1≤i≤n−1,

an(i, i + 1) = Pi

`=1 an−1(`, i + 1),1≤i≤n−2,

an(i, i + 2) = an(i, i + 1),1≤i≤n−2,

an(i, j) = Pj−1

k=i+1 an−1(i, k)

+Pi−1

a=1 Pi−a−1

c=0 Pj−c−2

b=a+1 i−a−1

can−c−2(a, b),

4≤i+ 3 ≤j≤n−1.

(18)

In this case, we state the recurrence formulas satisﬁed by the corresponding distribution

polynomials which will aid in translating (18) to functional equations as they are not too

lengthy. Deﬁne A+

n(v, w) = Pn−1

i=1 Pn

j=i+1 an(i, j)viwj,A−

n(v, w) = Pn

i=2 Pi−1

j=1 an(i, j)viwj

and An(v, w) = Pn

i=1 Pn

j=1 an(i, j)viwjfor n≥2. Clearly, An(v, w) = A+

n(v, w) + A−

n(v, w)

for all n≥2. Further, we deﬁne

Cn(v) =

n−2

X

i=1

an(i, i + 1)viand Bn(v, w) =

n−4

X

i=1

n−1

X

j=i+3

an(i, j)viwj.

Then (18) may be rewritten in terms of these distributions as

A−

n(v, w) = v

1−vAn−1(vw, 1) −vn+1

1−vAn−1(w, 1),

14 T. MANSOUR AND M. SHATTUCK

A+

n(v, w) = Bn(v , w) + w2(Cn(vw)−(vw)n−2An−2(1,1)) + wCn(vw) + wnAn−1(1, w),

Cn(v) = 1

vA+

n−1(1, v),

Bn(v, w) = w

1−w(Bn−1(v, w) + w2(Cn−1(vw)−An−3(1,1)(vw)n−3))

−wn

1−w(Bn−1(v, 1) + Cn−1(v)−An−3(1,1)vn−3)

+w3

1−w(Cn−1(vw)−An−3(1,1)vn−3wn−3)−wn

1−w(Cn−1(v)−An−3(1,1)vn−3)

+

n−4

X

i=1

n−1

X

j=i+3

i−1

X

a=1

i−a−1

X

c=0

j−c−2

X

b=a+1 i−1−a

can−c−2(a, b)viwj.

Deﬁne A±(x, v, w) = Pn≥2A±

n(v, w)xnand A(x, v , w) = Pn≥2An(v, w)xn. Further, deﬁne

B(x, v, w) = Pn≥2Bn(v , w)xnand C(x, v) = Pn≥2Cn(v)xn. Rewriting the preceding recur-

rences in terms of generating functions yields the following result.

Proposition 13. We have

A−(x, v, w) = v2wx2+vx

1−vA(x, vw, 1) −v2x

1−vA(vx, w, 1),

A+(x, v, w) = vw2x2−vw3x3+B(x, v, w) + w2(C(x, vw)−x2A(v wx, 1,1))

+wC(x, vw) + wxA(wx, v, 1),

where

C(x, v) = x

vA+(x, 1, v),

B(x, v, w) = wx(1 −v)

(1 −v−vwx)(1 −w)(B(x, v , w)−B(wx, v, 1))

+2(1 −vw)w3x

(1 −vw −vwx)(1 −w)C(x, vw)−2wx(1 −v)

(1 −v−vwx)(1 −w)C(wx, v)

+x2w2(1 −vwx)2

(1 −v−vwx)(1 −vw −vwx)(B(vwx

1−vwx ,1−vwx, 1) −B(x, 1−vwx, vw

1−vwx ))

+2x2w2(1 −vwx)2

(1 −v−vwx)(1 −vw −vwx)C(vwx

1−vwx ,1−vwx).

By mathematical programming, one may verify the following solution of the foregoing system

of functional equations.

Theorem 14. The generating function for the joint distribution of the ﬁrst and second letter

statistics on Sn(1243,1342) for n≥2is given by A(x, v, w) = A+(x, v , w) + A−(x, v, w),

where

A+(x, v, w)

=(1 −v+ (v2x−2v2+vx −x)w−vx(x−2)(v−1)w2)vw2x2

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw)) pv2w2x2−6vwx + 1

+(1 −v+ (6v2−4v−x(4v2−2v+ 1))w+xv(x(v2+ 4v−4) −2v2−6v+ 6)w2)vw2x2

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))

ON A CONJECTURE OF LIN AND KIM 15

−(x−2)(v−1)v3w5x4

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))

=vw2x2+w2v(vw +w+ 1)x3+w2v(2v2w2+ 2vw2+ 2vw + 2w2+w+ 1)x4+···

and

A−(x, v, w)

=(x(1 + v−2vx) + x(v2x−v2+vx −1)w−v(x−1)(vx −1)w2)v2wx2

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w)) pv2w2x2−6vwx + 1

+((3x−2)(w−1) + (−w2x2−3w2x−2wx2+ 3w2+ 2wx + 6x2−2w−3x)v)v2wx2

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))

+(w2x+wx2−w2+ 3wx −2x2−3w−2x+ 3 −(x−1)(w−1)vwx)v4w2x3

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))

=v2wx2+ (vw +v+ 1)v2wx3+ 2(v2w2+v2w+v2+vw +v+ 1)v2wx4+·· · .

Moreover, the generating functions counting the members of Sn(1243,1342) starting with an

ascent of size greater than two or of size exactly one and whose second letter is not nin either

case according to the ﬁrst and second letter statistics are given respectively by

B(x, v, w) = (vwx −2wx2+ 2wx +x−1)w2x2

2(1 −2vw −x(1 −vw))(1 −v−wx(2 −v)) pv2w2x2−6vwx + 1

+(1 −x+ (3vx −4v+ 2x−2)wx −(2x+v−6)vw2x2)w2x2

2(1 −2vw −x(1 −vw))(1 −v−wx(2 −v))

= 2vw4x5+ 2(4vw + 2w+ 1)vw4x6+ 217v2w2+ 10vw2+ 5vw + 4w2

+ 2w+ 1vw4x7+· ··

and

C(x, v) = −vx2(vx2−2vx −3x+ 2 + (x−2)√v2x2−6vx + 1)

2(1 −x−2v+vx)

=vx3+ (2v+ 1)vx4+ (6v2+ 3v+ 1)vx5+ (22v3+ 11v2+ 4v+ 1)vx6+· ·· .

Substituting w= 1 in the prior theorem and ﬁnding vx +A(x, v, 1), one obtains the following

result.

Corollary 15. The generating function for the distribution of the ﬁrst letter statistic on

Sn(1243,1342) for n≥1is given by

vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)

2(1 −v−2x+vx).

3.3. The case (1243,1324).We ﬁrst write a recurrence for an(i, j) when i<j.

Lemma 16. We have

(19) an(i, i + 1) = an−1(i, i + 1) +

i−1

X

a=1

i−a−1

X

c=0

i−c

X

b=a+1 i−a−1

can−c−2(a, b),1≤i≤n−2,

16 T. MANSOUR AND M. SHATTUCK

and

(20) an(i, j) = an−1(i, j )+

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1),3≤i+2 ≤j≤n−1,

with an(i, n) = an−1(i)for 1≤i≤n−1.

Proof. A similar proof may be given as in the prior two cases. Note that the formula for

an(i, n) is obvious since an nin the second position may clearly be removed. Let kdenote

the third letter of a member of Sn,i(1243,1324). If j=i+ 1 where 1 ≤i≤n−2, then either

k=i+ 2 or k < i, where clearly kmay be deleted in the former case. Assuming the latter, let

a, d denote the second leftmost ascent. Then we must have a+ 1 ≤d≤i+ 2, for otherwise a

1243 would occur. Thus, each letter prior to amay be deleted in this case and considering all

possible a,band c, where band care as before, implies formula (19). If i+ 2 ≤j < n, then

we have k=j+ 1 or k < i, the former leading to an−1(i, j) possibilities. On the other hand,

if k < i and a, d denotes the second ascent, then we must have d=j+ 1. To see this, note

that elements of [d+ 1, n] to the right of dmust form an increasing subsequence and thus

d < j < n would imply an occurrence of 1324 of the form ijxn where x∈[i+1, j−1]. Therefore

d=j+ 1 implies all letters prior to aagain may be deleted, resulting in a permutation that

starts a, j −c−1. Considering all possible aand cthen accounts for the double sum expression

in (20) and completes the proof.

From this, one may deduce the following further useful formula.

Lemma 17. If 1≤i≤n−3and i+ 2 ≤j≤n−1, then

(21) an(i, j) = an−1(1, j −1) + an−1(2, j −1) + · ·· +an−1(i−1, j −1) + an−1(i, j).

Proof. Note that (21) is clearly true on combinatorial grounds if i= 1 since the third letter

of a member of Sn(1243,1324) starting with 1, j where j < n must be j+ 1. So let i≥2 and

j∈[i+ 2, n −1]. Then by (20), we have

(22)

i−1

X

`=1

an−1(`, j −1) =

i−1

X

`=1 an−2(`, j −1) +

`−1

X

a=1

`−a−1

X

c=0 `−a−1

can−c−3(a, j −c−2)!.

If i≥3, then

i−1

X

`=2

`−1

X

a=1

`−a−1

X

c=0 `−a−1

can−c−3(a, j −c−2)

=

i−2

X

a=1

i−a−2

X

c=0

an−c−3(a, j −c−2)

i−1

X

`=a+c+1 `−a−1

c

=

i−2

X

a=1

i−a−2

X

c=0

an−c−3(a, j −c−2)i−a−1

c+ 1

=

i−2

X

c=1

i−c−1

X

a=1 i−a−1

can−c−2(a, j −c−1).

ON A CONJECTURE OF LIN AND KIM 17

Thus, the right-hand side of (22) is given by

i−1

X

`=1

an−2(`, j −1) +

i−2

X

c=1

i−c−1

X

a=1 i−a−1

can−c−2(a, j −c−1)

=

i−2

X

c=0

i−c−1

X

a=1 i−a−1

can−c−2(a, j −c−1)

=

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)

=an(i, j)−an−1(i, j ),

again by (20), which completes the proof.

Summarizing, we have the following recurrence relations satisﬁed by an(i, j):

an(i, j) = an−1(j),1≤j < i ≤n,

an(i, n) = an−1(i),1≤i≤n−1,

an(i, j) = Pi−1

`=1 an−1(`, j −1) + an−1(i, j ),

1≤i≤n−3 and i+ 2 ≤j≤n−1,

an(i, i + 1) = an−1(i, i + 1) + Pi−1

a=1 Pi

b=a+1 Pi−b

c=0 i−a−1

can−c−2(a, b),

1≤i≤n−2.

(23)

Deﬁne An(v, w), A±

n(v, w) and Cn(v) as in the previous subsection, but with Bn(v, w) now

given by Bn(v, w) = Pn−3

i=1 Pn−1

j=i+2 an(i, j)viwj. Translating (23) then yields the following

recurrences:

A−

n(v, w) =

n

X

i=2

i−1

X

j=1

an−1(j)viwj=v

1−vAn−1(vw, 1) −vn+1

1−vAn−1(w, 1),

A+

n(v, w) =

n−2

X

i=1

n−1

X

j=i+1

an(i, j)viwj+

n−1

X

i=1

an(i, j)viwn

=

n−3

X

i=1

n−1

X

j=i+2

an(i, j)viwj+w

n−2

X

i=1

an(i, i + 1)(vw)i+wnAn−1(v, 1)

=Bn(v, w) + wCn(vw) + wnAn−1(v , 1),

with

Bn(v, w) =

n−3

X

i=2

n−1

X

j=i+2

i−1

X

k=1

an−1(i, j −1)viwj+

n−3

X

i=1

n−1

X

j=i+2

an−1(i, j)viwj

=w

n−4

X

k=1

n−1

X

j=k+3

j−2

X

i=k+1

an−1(i, j −1)viwj+

n−4

X

i=1

n−2

X

j=i+2

an−1(i, j)viwj+

n−3

X

i=1

an−2(i)viwn−1

=wv

1−vBn−1(v, w)−w

1−vBn−1(1, vw) + Bn−1(v , w) + wn−1An−2(v, 1)

−wn−1vn−2An−3(1,1)

18 T. MANSOUR AND M. SHATTUCK

and

Cn(v) = Cn−1(v) + vn−2An−3(1,1) +

n−2

X

i=1

i−1

X

a=1

i

X

b=a+1

i−b

X

c=0 i−a−1

can−2−c(a, b)vi

=Cn−1(v) + vn−2An−3(1,1) +

n−3

X

a=1

n−2

X

b=a+1

n−2−b

X

c=0

n−2

X

i=b+ci−a−1

can−2−c(a, b)vi.

Deﬁne A(x, v, w), A±(x, v , w) and C(x, v) as in the previous subsection, with B(x, v, w) =

Pn≥2Bn(v, w)xnper the new deﬁnition for Bn(v, w). From the preceding recurrences, we

obtain the following system of functional equations.

Proposition 18. We have

A−(x, v, w) = v2wx2+vx

1−vA(x, vw, 1) −v2x

1−vA(vx, w, 1),

A+(x, v, w) = vw2x2+B(x, v, w) + wC(x, vw) + wxA(wx, v, 1),

where

B(x, v, w) = wx

1−v(vB(x, v, w)−B(x, 1, v w)) + xB(x, v, w) + wx2A(wx, v, 1)

−vw2x3A(v wx, 1,1) −v2w3x4,

C(x, v) = vx3(1 + vx)

1−x+vx3

1−xA(vx, 1,1)

+x2

(vx +v−1)(1 −x)(vA+(vx

1−vx ,1−vx, 1) −(1 −vx)A+(x, 1−vx, v

1−vx )).

By programming, one may verify that the solution of the foregoing system is given as follows.

Theorem 19. We have A(x, v, w) = A+(x, v, w) + A−(x, v, w), where

A+(x, v, w)

=(1 −v+ (v2x−2v2+vx −x)w−vx(x−2)(v−1)w2)vw2x2

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw)) pv2w2x2−6vwx + 1

+(1 −v+ (6v2−4v−x(4v2−2v+ 1))w+xv(x(v2+ 4v−4) −2v2−6v+ 6)w2)vw2x2

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))

−(x−2)(v−1)v3w5x4

2(1 −v+wx(v−2))(1 −2vw −x(1 −vw))

=vw2x2+w2v(vw +w+ 1)x3+w2v(2v2w2+ 2vw2+ 2vw + 2w2+w+ 1)x4+···

and

A−(x, v, w)

=(x(1 + v−2vx) + x(v2x−v2+vx −1)w−v(x−1)(vx −1)w2)v2wx2

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w)) pv2w2x2−6vwx + 1

+((3x−2)(w−1) + (−w2x2−3w2x−2wx2+ 3w2+ 2wx + 6x2−2w−3x)v)v2wx2

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))

+(w2x+wx2−w2+ 3wx −2x2−3w−2x+ 3 −(x−1)(w−1)vwx)v4w2x3

2(1 −vw −x(2 −vw))(1 −w−vx(2 −w))

ON A CONJECTURE OF LIN AND KIM 19

=v2wx2+ (vw +v+ 1)v2wx3+ 2(v2w2+v2w+v2+vw +v+ 1)v2wx4+·· · .

Moreover,

B(x, v, w)

=−vw2x2(wx2−2wx −x+ 1)

2(vwx −2vw −x+ 1)(vwx −2wx −v+ 1) pv2w2x2−6vwx + 1

−vw2x2(vw2x3−2vw2x2−vwx2+ 3vwx −3wx2+ 2wx +x−1)

2(vwx −2vw −x+ 1)(vwx −2wx −v+ 1)

=vw3x4+w3v(3vw + 2w+ 1)x5+w3v(11v2w2+ 8vw2+ 4vw + 4w2+ 2w+ 1)x6+···

and

C(x, v) = −vx2(vx2−2vx −3x+ 2 + (x−2)√v2x2−6vx + 1)

2(1 −x−2v+vx).

Taking w= 1 in the prior theorem yields the following result.

Corollary 20. The generating function for the distribution of the ﬁrst letter statistic on

Sn(1243,1324) for n≥1is given by

vx(2 −3v−3x+ 3vx) + vx(v+x−vx)(vx +√1−6vx +v2x2)

2(1 −v−2x+vx).

4. Appendix

In this section, we provide proofs of the formulas obtained in Proposition 8 and Theorem 9.

We omit the proofs of the comparable steps in the derivations of Theorems 14 and 19.

Proof of Proposition 8:

Clearly, A+(x, v, w) = wC(x, vw) + w2D(x, vw) + B(x, v, w), by the deﬁnitions. Translating

(11)-(14) in terms of generating functions yields

A−(x, v, w) = X

n≥2

n

X

a=2

a−1

X

b=1

an−1(b)vawbxn

=X

n≥2

n−1

X

b=1

an−1(b)vb+1 −vn+1

1−vwbxn

=v2wx2+vx

1−vA(x, vw, 1) −v2x

1−vA(vx, w, 1),

wC(x, vw)−vw2x2A(vwx, 1,1) −vw2x2−v2w3x3

=wxC(x, vw) + X

n≥2

n−2

X

i=1

i−1

X

a=1

i−a−1

X

c=0

i−c

X

b=a+1 i−a−1

can−c−2(a, b)viwi+1xn

=wxC(x, vw) + wX

a≥1X

c≥0X

i≥1X

n≥i+c+a+2

i+a

X

b=a+1 i+c−1

can−c−2(a, b)(vw)i+a+cxn

=wxC(x, vw) + wX

a≥1X

i≥1X

n≥a+i

i+a

X

b=a+1

an(a, b)(vw)i+axn+2

(1 −vwx)i

20 T. MANSOUR AND M. SHATTUCK

=wxC(x, vw)

+w

vwx +vw −1X

n≥2

n−1

X

a=1

n

X

b=a+1

an(a, b)( xn+2(vw)n+1

(1 −vwx)n−a−xn+2 (vw)b

(1 −vwx)b−a−1)

=wxC(x, vw)

+vw2x2

vwx +vw −1A+(vwx

1−vwx ,1−vwx, 1) −(1 −vwx)wx2

vwx +vw −1A+(x, 1−vwx, vw

1−vwx ),

w2D(x, vw)−w2x2A(v wx, 1,1) + v2w4x4

=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)

+w2X

n≥3

n−3

X

i=1

i−1

X

a=1

i−1−a

X

c=0

i−c+1

X

b=a+1 i−1−a

can−2−c(a, b)(vw)ixn

=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)

+w2X

a≥1X

c≥0X

i≥1X

n≥i+a+3

i+a+1

X

b=a+1 i+c−1

can−2(a, b)(vw)i+a+cxn+c

=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)

+w2x2X

a≥1X

n≥a+2

n

X

b=a+2

an(a, b)(vw)i+axn

(1 −vwx)i+w2x2X

a≥1X

n≥a+2

n−a−1

X

i=1

an(a, a + 1) (vw)i+axn

(1 −vwx)i

=w2x(C(x, vw)−vwx2A(vwx, 1,1)) + w2xD(x, vw)

+w2x2(1 −vwx)

vwx +vw −1A+(vwx

1−vwx ,1−vwx, 1) −wx2(1 −vwx)2

v(vwx +vw −1)A+(x, 1−vwx, vw

1−vwx )

−w2x2C(x, vw),

and

B(x, v, w)

=wxB(x, v, w) + w3xD(x, vw)

+X

n≥3

n−3

X

i=1

n

X

j=i+3

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−2)viwjxn

+X

n≥3

n−3

X

i=1

n

X

j=i+3(1 −δj,n)(an−1(i, j )viwjxn

+

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)viwjxn

=wxB(x, v, w) + w3xD(x, vw)

+X

n≥3

n−3

X

i=1

n

X

j=i+3

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−2)viwjxn

ON A CONJECTURE OF LIN AND KIM 21

+X

n≥3

n−3

X

i=1

n−1

X

j=i+3 an−1(i, j)viwjxn+

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)viwjxn!

=wxB(x, v, w) + w3xD(x, vw)

+X

a≥1X

n≥a+2

n

X

j=a+2

j−a−1

X

i=1

an(a, j)vi+awj+2 xn+2

(1 −vwx)i

+X

n≥3

n−3

X

i=1

n−1

X

j=i+3 an−1(i, j)viwjxn+

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)viwjxn!

=wxB(x, v, w) + w3xD(x, vw)

+w2x2(1 −vwx)

vwx +v−1A+(x, 1−vwx, vw

1−vwx )−vw2x2

vwx +v−1A+(x, v , w)

+X

n≥3

n−3

X

i=1

n−1

X

j=i+3 an−1(i, j)viwjxn+

i−1

X

a=1

i−a−1

X

c=0 i−a−1

can−c−2(a, j −c−1)viwjxn!

=wxB(x, v, w) + w3xD(x, vw)

+w2x2(1 −vwx)

vwx +v−1A+(x, 1−vwx, vw

1−vwx )−vw2x2

vwx +v−1A+(x, v , w)

+xB(x, v, w) + 1

vwx +v−1X

a≥1X

n≥a+3

n

X

j=a+3

an(a, j)( vj−1wj+1 xnn+2

(1 −vwx)j−2−a−va+1 wj+1xnn+2)

=wxB(x, v, w) + w3xD(x, vw)

+w2x2(1 −vwx)

vwx +v−1A+(x, 1−vwx, vw

1−vwx )−vw2x2

vwx +v−1A+(x, v , w)

+xB(x, v, w) + wx2(1 −vwx)2

v(vwx +v−1) B(x, 1−vwx, vw

1−vwx )−vwx2

vwx +v−1B(x, v , w).

Combining the results above gives Proposition 8.

Proof of Theorem 9:

By Proposition 8, one may express Bin terms of the generating functions A, C, D and C, D

in terms of A. Substituting these relations into the last equation from Proposition 8, one

obtains

vx −wx −v−x+ 1

vwx +v−1A+(x, v , w)(24)

=w2x2(vwx −v−1)A+(v wx

1−vwx ,1−vwx, 1)

vwx +vw −1

+vwx2(w2−1)(vwx −1)A+(x, 1−vwx, vw

1−vwx )

(vwx +v−1)(vwx +vw −1)

+w2x2(vwx −v−1)A(v wx, 1,1) + x2vw2(vw2x2−vwx −1).

22 T. MANSOUR AND M. SHATTUCK

Replacing wby w/v in (24), we have

v2x−wx −v2−vx +v

v(wx +v−1) A+(x, v, w/v)

(25)

=w2x2(wx −v−1)A+(wx

1−wx ,1−wx, 1)

v2(wx +w−1) +wx2(w2−v2)(wx −1)A+(x, 1−wx, w

1−wx )

v2(wx +v−1)(wx +w−1)

+w2x2

v2(wx −v−1)A(wx, 1,1) + x2w2

v2(w2x2−vwx −v).

We now seek to determine a formula for A+(x, v, w) using (25). First note that by (2) and

the main result from [8], we have

A(x, 1,1) = 1−3x−√1−6x+x2

2,(26)

which implies A+(x, v, 1) may be determined independently of A−(x, v, 1) since Aoccurs

in (25) only through the A(wx, 1,1) term. Suppose for a moment that A(x, v, 1) is as in

Corollary 10. Then the second equation in Proposition 8 above at w= 1, together with the

fact A(x, v, 1) = A+(x, v, 1) + A−(x, v, 1), gives a linear system of equations in the quantities

A+and A−. This yields

A+(x, v, 1) = vx2(1 + v−vx)

2(vx −v−2x+ 1) p1−6vx +v2x2−vx2(v2x2−v2x−4vx + 3v−1)

2(vx −v−2x+ 1) ,(27)

which we will assume for now to aid in solving (25).

Then taking v=v0=1−x+√1−2(1+2w)x+(1+4w)x2

2(1−x)in (25), and using (26) and (27), implies

A+(x, 1−wx, w

1−wx )

=(1 −x)(wx +w−1)

4(wx −w−2x+ 1)(wx −1) p1−6wx +w2x2p(1 −x)(1 −x−4vwx)

+1−w+ (2w2−w−2)x−(w−1)(2w2+ 4w+ 1)x2+w(2w2−2w−1)x3

4(wx −w−2x+ 1)(wx −1) p1−6wx +w2x2

−(wx +w−1)(wx2−wx −3x+ 1)

4(wx −w−2x+ 1)(wx −1) p(1 −x)(1 −x−4vwx)

+w−1−(3w2−4)x+ (8w3+w2−9w−3)x2

4(wx −w−2x+ 1)(wx −1)

+−w(2w3+ 8w2−9w−4)x3+w2(2w2−2w−1)x4

4(wx −w−2x+ 1)(wx −1) .

Substituting this expression into (24), and using (26) and (27), we obtain

A+(x, v, w)(28)

=(w2−1)(1 −x)vwx2√1−6vwx +v2w2x2p(1 −x)(1 −x−4vwx)

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

+(1 −x−2vwx)vw(1 −x)x2√1−6vwx +v2w2x2

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

ON A CONJECTURE OF LIN AND KIM 23

+(1 −2v2x2+ 4v2x−2v2−x2−2x−2vwx(1 −x))vw3x2√1−6vwx +v2w2x2

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

+(1 −w2)(vwx2−vwx −3x+ 1)vwx2p(1 −x)(1 −x−4vwx)

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

+(3(w2−1)x2+ 4(1 −2w)x−w2+ 4w−1)vwx2

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

−((w2−1)x3+ 8(w2+ 1)x2−(7w2+ 4w+ 11)x+ 4w+ 4)v2w2x2

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1)

−2((1 −x)(w2x2+x2+ 3x−3) + vwx(1 −x)2)v3w3x2

4(vwx −vw −2x+ 1)(vx −wx −v−x+ 1) .

One may verify using programming that this expression for A+(x, v, w) indeed satisﬁes (24)

and (27) and thus is the solution of (24) that is sought. We may now determine A−(x, v, w).

By the second equation in Proposition 8, we have

A−(x, v, 1) = v2x2+vx

1−vA(x, v, 1) −v2x

1−vA(vx, 1,1)

=v2x2+vx

1−v(A+(x, v, 1) + A−(x, v, 1)) −v2x

1−vA(vx, 1,1).

Solving for A−(x, v, 1) in this last equation, and using (26) and (27), yields

A−(x, v, 1) = v2x(x−1)(vx2−vx −3x+ 1 + (x−1)√1−6vx +v2x2)

2(vx −v−2x+ 1) .

Hence, by Proposition 8, we get the following explicit formula for A−(x, v, w):

A−(x, v, w)

(29)

=((1 + v−2vx)x+ (v2x−v2+vx −1)wx −(1 −x)(1 −vx)vw2)v2wx2

2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1) p1−6vwx +v2w2x2

+(6vx2−3(v+ 1)x+ 2 −(2v2x3+ 2v(v+ 1)x2−(3v2+ 2v+ 3)x+ 2v+ 2)w)v2wx2

2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1)

+(v2x3−v2x2+vx3+ 3vx2−3vx −x2−3x+ 3 −(1 −x)(1 −vx)vwx)v3w3x2

2(vwx −2vx −w+ 1)(vwx −vw −2x+ 1) .

Combining the formulas for A+(x, v, w) and A−(x, v, w) yields the formula for A(x, v, w).

References

[1] G. E. Andrews and S. Chern, A proof of Lin’s conjecture on inversion sequences avoiding patterns of

relation triples, J. Combin. Theory Ser. A 179 (2021), Art. 105388.

[2] J. Bonin, L. Shapiro and R. Simion, Some q-analogues of the Schr¨oder numbers arising from combinatorial

statistics on lattice paths, J. Statist. Plann. Inference 34 (1993), 35–55.

[3] W. Cao, E. Y. Jin and Z. Lin, Enumeration of inversion sequences avoiding triples of relations, Discrete

Appl. Math. 260 (2019), 86–97.

[4] S. Corteel, M. Martinez, C. D. Savage and M. Weselcouch, Patterns in inversion sequences I, Discrete

Math. Theor. Comput. Sci. 18 (2016), #2.

[5] S. Gire, Arbres, permutations `a motifs exclus et cartes planaire: quelques probl`emes algorithmiques et

combinatoires, Ph.D. These, University of Bordeaux, 1993.

24 T. MANSOUR AND M. SHATTUCK

[6] Q. Hou and T. Mansour, Kernel method and linear recurrence system, J. Comput. Appl. Math. 216(1)

(2008), 227–242.

[7] S. Kitaev, Patterns in Permutations and Words, Springer-Verlag, Berlin, 2011.

[8] D. Kremer, Permutations with forbidden subsequences and a generalized Schr¨oder number, Discrete Math.

218 (2000), 121–130.

[9] Z. Lin, Restricted inversion sequences and enhanced 3-noncrossing partitions, European J. Combin. 70

(2018), 202–211.

[10] Z. Lin, Patterns of relation triples in inversion and ascent sequences, Theoret. Comput. Sci. 804 (2020),

115–125.

[11] Z. Lin and D. Kim, Reﬁned restricted inversion sequences, arXiv:1706.07208v2, accepted, Ann. Comb.

(2021).

[12] Z. Lin and S. H. F. Yan, Vincular patterns in inversion sequences, Appl. Math. Comput. 364 (2020), Art.

124672.

[13] T. Mansour and M. Shattuck, Pattern avoidance in inversion sequences, Pure Math. Appl. (PU.M.A.) 25

(2015), 157–176.

[14] E. Pergola and R. A. Sulanke, Schr¨oder triangles, paths and parallelogram polyominoes, J. Integer Seq.

1(1998), Art. 98.1.7.

[15] J. L. Ram´ırez and V. F. Sirvent, Generalized Schr¨oder matrix and its combinatorial interpretation, Linear

Multilinear Algebra 66 (2018), 418–433.

[16] D. G. Rogers, A Schr¨oder triangle: Three combinatorial problems, Combinatorial Mathematics V, Lecture

Notes in Mathematics 622, Springer, Berlin, 1977, 175–196.

[17] N. J. Sloane, The On-Line Encyclopedia of Integer Sequences, http://oeis.org, 2010.

[18] J. West, Generating trees and the Catalan and Schr¨oder numbers, Discrete Math. 146 (1995), 247–262.

[19] J. West, Generating trees and forbidden subsequences, Discrete Math. 157 (1996), 363–374.

[20] C. Yan and Z. Lin, Inversion sequences avoiding pairs of patterns, Discrete Math. Theor. Comput. Sci.

22(1) (2020), Art. 23.

Department of Mathematics, University of Haifa, 3498838 Haifa, Israel

E-mail address:tmansour@univ.haifa.ac.il

Department of Mathematics, University of Tennessee, 37996 Knoxville, TN

E-mail address:mshattuc@utk.edu