Content uploaded by Olanrewaju Ismail Shittu

Author content

All content in this area was uploaded by Olanrewaju Ismail Shittu on Apr 07, 2014

Content may be subject to copyright.

European Journal of Scientific Research

ISSN 1450-216X Vol.30 No.3 (2009), pp.409-416

© EuroJournals Publishing, Inc. 2009

http://www.eurojournals.com/ejsr.htm

Comparison of Criteria for Estimating the Order of

Autoregressive Process: A Monte Carlo Approach

Shittu. O.I

Department of Statistics, University of Ibadan, Ibadan, Nigeria

E-mail: shiittu.olanrewaju@gmail.com

Asemota. M.J

Department of Statistics, University of Ibadan, Ibadan, Nigeria

Abstract

This paper compares the performance of model order determination criteria in terms

of selecting the correct order of an Autoregressive model in small and large samples using

the simulation method. The considered criteria are the Akaike information criterion (AIC);

Bayesian information criterion (BIC) and the Hannan − Quinn criterion (HQ). Eight series

of sizes N=50 and N=200 were generated with 50 replicates each under consistent

assumptions and parameter specifications.

Our results shows that Bayesian information criteria (BIC) performs best in terms of

selecting the correct order of an Autoregressive model for small samples irrespective of the

AR structure, Hannan Quinn criteria can be said to perform best in large sample. Even

though the AIC has the least performance among the criteria considered, it appears to be

the best in terms of the closeness of the selected order to the true value. Our results

compares favorably with that of Lutkepohl (1985) and Benedikt Pötscher (1991).

Keywords: Monte Carlo; Information, Criteria; Performance; Simulation.

1. Introduction

It is well known that most economic and financial series follow the Autoregressive Moving average

[ARMA(p,q)] model, and more often than not the autoregressive [AR(p)] model where p and/or q are

the order of the model. However, determination of the correct order p,q has been a source of serious

concern to analysts over the years, since inappropriate order selection may result into inconsistent

estimate of parameters if p< true value or may not be consistent and increase in the variance of the

model if p>true, (Shibata, 1976).

In recent years there has been a substantial literature on this problem and different criteria have

been proposed to aid in choosing the order of the ARMA(p,q) process. These criteria are based on

theoretical considerations that provide only asymptotic properties of the resulting estimators. The

practitioner, however, usually faces the problem of making a choice on the basis of a limited data set.

Among the criteria considered are the Akaike information criteria (AIC); Bayesian Information

criteria (BIC), Hannan Quinne criteria (HQ); Carlos Information criteria (CIC) and others.

This paper therefore compares the performance of the commonly used information criteria such

as: Akaike Information Criteria (AIC), Bayesian Information Criteria (BIC), and Hannan – Quinn

Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte

Carlo Approach 410

(HQ) Criterion, for the Autoregressive AR(p) model using the Monte Carlo simulation method for

small and large samples.

2. Review of Literature

The information criterion has been widely used in time series analysis to determine the appropriate

order of a model. The information criteria are often used as a guide in model selection. The goal of any

order selection procedure is to estimate the order p for an AR model on the basis of n successive

observations from a time series X(t) while the notion of an information criterion is to provide a

measure of information in terms of the order of the model, that strikes a balance between these

measures of goodness of fit and parsimonious specification of the model. The Kullback – Leibler

quantity of information contained in a model is the distance from the “true” model and is measured by

the log likelihood function.

Several criteria are used for this purpose, in particular, we discuss the AIC (Akaike, 1974), BIC

(Rissanen, 1978; Scwarz, 1978) and HQ (Hannan and Quinn, 1979) amongst others.

All these criteria aim at minimizing the residual sum of squares and impose a penalty for

including an increasingly large number of regressors (lag values).

2.1. Akaike Information Criterion

The Akaike Information Criterion (AIC), (Akaike, 1974) is an objective measure of model suitability

which balances model fit and model complexity. Considering a stationary time series {Xt}, t = 1, 2, …

N, the Akaike information criteria consist of minimizing the function:

K(p) = N loge2

p

σ

+ pc(n) (3.1)

p∈P = { 0, 1, 2, … m}

where 2

p

σ

is the estimated residual variance for a fitted AR (p) model, c(n) is a penalty term, N is the

number of observations and m is a pre – determined upper autoregressive order.

To obtain

AIC (p) = 2

log pe

N

σ

+ 2p (3.2)

p ∈ P= { 0, 1, 2, … m}

where

2

11

21∑∑

==

−⎥

⎦

⎤

⎢

⎣

⎡−= N

t

p

jjtjtp XX

N

φσ

or more compactly as; ∑

=22 1tp N

εσ

selection of the chosen model is then made by considering the minimum

AIC = min {AIC(p)}, that is the model with smallest AIC is chosen.

Details of the proof can be found in Akaike (1974) and Shibata (1976).

One advantage of AIC is that it is useful for not only in – sample data but also out–of–sample

forecasting performance of a model. In – sample forecasting essentially tells us how the chosen model

fits the data in a given sample while the out–of–sample forecasting is concerned with determining how

a fitted model forecasts future values of the regressed, given the values of the repressors. It is also

useful for both nested and non-nested models. The outstanding demerit of this criterion is that of

inconsistency. The procedure has been criticized because it is inconsistent and tends to over– fit a

model, Shibata(1976) showed this for autoregressive model and Hannan(1982) for ARMA models.

411 Shittu. O.I and Asemota. M.J

2.2. Bayesian Information Criteria

The Bayesian information criterion is obtained by replacing the non-negative function c(n) in (2.2) by

loge(N). Hence, we have

BIC(p) = NpN epe loglog 2+

∧

σ

(3.3)

p ∈ P = {0, 1, 2, … m}

where ∧2

p

σ

is obtained as above and the appropriate model is obtained as that which minimizes the BIC

(p) above, that is, min (BIC(p))

Details of the discussion can be found in Rissanen (1978); Schwarz (1978) and Stone (1979).

The BIC imposes a harsher penalty than AIC, as its obvious from comparing (3.2) to (3.3). An

important advantage of BIC is that for a wide range of statistical problems, it is order consistent (i.e.

when the sample size grows to infinity, the probability of choosing the right model converges to unity)

leading to more parsimonious models. Like the AIC, the lower the value of BIC, the better the model.

Like AIC, BIC can be used to compare in–sample or out–of–sample forecasting performance of a

model.

2.3. Hannan – Quinn Criterion

The Hannan-Quinn criterion for identifying an autoregressive model denoted by HQ (p) was

introduced by Hannan and Quinn (1979). The adjusted version of it can also be applied to regression

models, Al-Subaihi (2007). It is obtained by replacing the non – negative penalty function c(n) in

equation (3.2) by ClogClogeN.

Thus, we have

HQ(p) = Nloge ∧2

p

σ

+ p C ln Cloge(N) (3.4)

p ∈ P = {0, 1, … m}.

where C is a constant, C > 2. For practical purpose, we set C = 2. The best model is the model that

correspond to minimum HQ i.e min (HQ (p))

The order selection procedure presented above have the advantage of being objective and

automatic, but it over-fit when the sample size is small.

Detailed discussion on this can be found in Hannan-Quinn (1979); McQuarrie and Tsai (1998).

Having noticed the limitations of these criteria, the aim of this study is to examine the veracity

of these claims and compare their performance using the Monte Carlo study.

3. Simulation Study

The basic theory behind random number generation with computers offers a simple example of Monte

Carlo simulation to understand the properties of different statistics computed from sample data. Monte

Carlo methods comprise that branch of experimental mathematics, which is concerned with utilization

of random normal deviates. The random deviates are generated using Microsoft Excel random number

generator to have zero mean and unit standard deviation, N(0,1). The routine generates realization for a

given AR structure.

In order to contrast the performance of the order identification criteria, simulation study was

conducted using a wide range of autoregressive (AR) processes with different characteristics.

The following assumptions were made. The random numbers follow a standard normal

distribution with mean zero and variance unity. Data were generated for samples of N = 50 and N =

200 with 50 replications each. Autoregressive models of orders p = 0, 1, 2, 3, and 4 were generated

with a maximum of 5. The parameters of all the generated series in the simulation study are chosen so

that the series are stationary that is the roots fall outside the unit circle. The autoregressive structures

p1 to p8 and their parameters are given in the table below:

Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte

Carlo Approach 412

Table 1: Data Generating Processes

Process AR Structure Parameter Values

AR (1)

P1 ROOT: 10 φ = 0.1

AR (1)

P2 ROOT: - 1.11 φ = - 0.9

AR (2) φ1 = 0.7

P3 ROOT: 5, 2 φ2 = - 0.1

AR (2) φ1 = 0.8

P4 ROOT: - 1.45, - 9.22 φ2 = 0.075

AR (3) φ1 = 0.8, φ2 = 0.3

P5 ROOT: 2.29, 1.13, - 1.93 φ3 = - 0.2

AR (3) φ1 = 0.9, φ2 = 0.5

P6 ROOT: - 1.30, 1.07+I, 1.07 – i φ3 = - 0.6

AR (4) φ1 = φ2 = φ3 = 0

P7 ROOT: ± 1.19 four-times φ4 = 0.5

AR (4) φ1 = 0.9, φ2 = 0.7

P8 ROOT: 1.11, 1.142, 2.00, 3.33 φ3 = 0.5, φ4 = 0.3

From Table 1 above, we have two processes for each of the structures AR(1), AR(1), AR(3)

and AR (4). Process p1 is an autoregressive structure of order (1) having its root located far away from

the unit circle, while process p2 is an AR (1) structure with its root close to the unit circle. Process p3 is

an AR (2) structure with the two roots located further away from the unit circle, process p4 is also AR

(2) structure, with one of its roots very close to the unit circle.

Process p5 is an AR (3) structure with two of its roots very close to the unit circle, while process

p6 is also an AR (3) structure wit two of its roots being complex while the other falls barely outside the

unit circle. This scenario represents a situation of a mixture of real and complex roots.

Process p7 is an AR(4) structure with repeated/multiple roots, while process p8 is also an AR (4)

structure which has two of its roots close to the unit circle while the other two roots are some distance

from the unit circle.

For each of the model structure, an autoregressive model was fit the model order were

examined using the criteria AIC, BIC and HQ using the E – views software package.

The performance criterion is that, the information criterion with the highest number of cases (or

percentage) of selecting the correct order of the given AR structure is considered to be the best.

4. Results and Discussions

After fitting the models to the generated data, the number of times each identification criteria (AIC,

BIC and HQ) was able to accurately identity the correct order of a given AR structure were counted

and frequencies obtained.

In measuring the performance of the generated models, the criterion that has the highest

number of cases (or percentage frequency) of selecting the correct order of the given AR structure is

considered to be the best criterion. The results of the analysis were tabulated different criterion for N =

50 in able 2 and for N = 200 in Table 3.

413 Shittu. O.I and Asemota. M.J

Table 2: Frequency And Percentage Distribution Of From Simulated Data When N = 50 And R = 50

Order Selected

AR Processes Criterion 0 1 2 3 4 5

AIC -

40(80%) 8(16%) 2(4%) - -

BIC -

45(90%) 4(8%) 1(2%) - -

P1 HQ -

42(84%) 6(12%) 2(4%) - -

AIC 1(2%) 36(72%) 10(20%) 3(6%) - -

BIC -

43(86%) 5(10%) 2(4%) - -

P2 HQ 1(2%) 40(80%) 8(16%) 1(2%) - -

AIC 2(4%) 8(16%) 35(70%) 5(10%) - -

BIC 1(2%) 7(14%) 36(72%) 6(12%) - -

P3 HQ -

13(26%) 30(60%) 7(14%) - -

AIC 1(2%) 2(4%) 43(86%) 4(8%) - -

BIC 1(2%) 3(6%) 44(88%) 2(4%) - -

P4 HQ 1(2%) 3(6%) 42(84%) 4(8%) - -

AIC -

2(4%) 4(8%) 38(76%) 6(12%) -

BIC - -

6(12%) 41(82%) 3(6%) -

P5 HQ - -

7(14%) 42(84%) 1(2%) -

AIC -

1(2%) 5(10%) 40(80%) 4(8%) -

BIC - -

3(6%) 46(92%) 1(2%) -

P6 HQ - -

4(8%) 44(88%) 2(4%) -

AIC - -

3(6%) 4(8%) 40(80%) 3(6%)

BIC - - -

1(2%) 48(96%) 1(2%)

P7 HQ - - -

2(4%) 47(94%) 1(2%)

AIC - - -

4(8%) 44(88%) 2(4%)

BIC - - -

3(6%) 46(92%) 1(2%)

P8 HQ - -

1(2%) 6(12%) 41(82%) 2(4%)

From the above table for small sample (N = 50), it can be observed that BIC criterion performs

best in terms of the percentage number of correct order identified in various AR models regardless of

the parameterization and sample size. This is closely followed by Hannan-Quinn criterion.

Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte

Carlo Approach 414

Table 3: Frequency And Percentage Distribution Of Order From Simulated Data When N = 200 And R = 50

Order Selected

AR Processes Criterion 0 1 2 3 4 5

AIC 1(2%) 40(80%) 5(10%) 4(8%) - -

BIC -

44(88%) 3(6%) 3(6%) - -

P1 HQ 1(2%) 46(92%) 2(4%) 1(2%) - -

AIC 2(4%) 40(80%) 6(12%) 2(4%) - -

BIC -

45(90%) 3(6%) 2(4%) - -

P2 HQ 1(2%) 43(86%) 5(10%) 1(2%) - -

AIC 1(2%) 8(16%) 38(76%) 3(6%) - -

BIC -

3(6%) 46(92%) 1(2%) - -

P3 HQ -

3(6%) 42(84%) 5(10%) - -

AIC -

7(14%) 42(84%) 1(2%) - -

BIC -

5(10%) 45(90%) - - -

P4 HQ -

2(4%) 47(94%) 1(2%) - -

AIC -

1(2%) 4(8%) 41(82%) 4(8%) -

BIC - -

1(2%) 48(96%) 1(2%) -

P5 HQ - -

2(4%) 45(90%) 3(6%) -

AIC -

1(2%) 3(6%) 40(80%) 6(12%) -

BIC -

1(2%) 3(6%) 44(88%) 2(4%) -

P6 HQ -

2(4%) 4(8%) 43(86%) 1(2%) -

AIC - -

2(4%) 4(8%) 41(82%) 3(6%)

BIC -

1(2%) 2(4%) 2(4%) 42(84%) 3(6%)

P7 HQ - - -

2(4%) 45(90%) 3(6%)

AIC - - -

6(12%) 40(80%) 4(8%)

BIC - - -

1(2%) 47(94%) 2(4%)

P8 HQ - - -

2(4%) 45(90%) 3(6%)

From Table 3, we found that the BIC performed better in five out of the eight AR processes

considered. The HQ followed closely with even a better performance in three of the AR process.

The AIC performance was the least in all the processes considered. From these results, it

appears that the performance of HQ criterion is slightly improved with increase in sample size, this is

not surprising in the sense that, in principle, as the sample size increased, convergence is expected to

occur for all criteria especially for Hannan-Quinn criterion. however both the AIC and HQ exhibit

highest rate of inconsistency in their estimation as they have higher rate of over estimation or under

estimation of the order of the models. We also found that both the AIC and HQ exhibit highest rate of

inconsistency in their estimation as they have higher rate of over estimation or under estimation of the

order of the models. This result agrees with that of Poskitt (1994) and Salau (2002).

The closeness of the value of the selected order to the true value for each of the information

criteria is given in the table below.

Table 4: Average Number Of Ar Order (P) Selected For Simulated Data

Model P1 P = 1 P2 P = 1 P3 P = 2 P4 P = 2 P5 P = 3 P6 P = 3 P7 P = 4 P8 P = 4

N 50 200 50 200 50 200 50 200 50 200 50 200 50 200 50 200

Observed Average

value of p 1.1 1.2 1.1 1.3 1.8 1.9 1.8 1.9 2.9 3.0 2.9 3.0 3.9 4.0 3.9 4.0

AIC 4 4 0 6 6 6 4 8 6 6 4 2 6 0 8 6

BIC 2 8 8 4 4 6 4 0 4 0 6 2 8 0 6 2

HQ 6 2 8 2 8 4 8 8 8 2 6 6 8 2 8 2

415 Shittu. O.I and Asemota. M.J

Table 4 displays the average order selected by the criteria as compared with the specified ‘true’

orderHence on the average, the BIC is seen to outperform both the HQ and AIC, regardless of the

sample size and the AR structures considered, with the AIC having the least performance.

5. Summary and Conclusion

Based on the result of our simulation study, we conclude that the Bayesian information criteria (BIC)

performs best in terms of selecting the correct order of an Autoregressive model for small samples

irrespective of the AR structure, parameter values and nature of the roots of the resulting polynomial.

The Hannan Quinn criteria can be said to perform best in large sample as it performed best in

five of the eight cases considered for large samples and its capacity for estimating the true order the

next best on the remaining cases. The performance of AIC improved with increase in sample size but

still had the poorest performance in all the eight processes. It can also be observed that both AIC and

HQ were mostly inconsistent in their estimation for both small and large samples.

It was also found that orders selected by HQ were closes to the true value for small sample size

while BIC was closest for large samples. Even though the AIC has the least performance among the

criteria considered, it appears to be the best in terms of the closeness of the selected order to the true

value. These research findings are quite in agreement with the work of Lutkepohl (1985) for vector

autoregressive process (VAR) and the works of Benedikt Pötscher (1991) on ARMA models.

However, according to Al-Subaihi(2002), it is important for an investigator not to depend

entirely on an information criterion in evaluating the order of a model because non of them works well

under all conditions. Their performance depends on sample size, number of dependent and independent

variable as well as the correlation between them in the case of multiple regression models.

A researcher needs to evaluate the ‘good’ model using various model order criteria bearing in

mind the conditions mentioned above.

Acknowledgement

This paper is partly funded by Senate research grant number SRG/FSC/2006/2A of the University of

Ibadan, Nigeria.

Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte

Carlo Approach 416

References

[1] Akaike, H (1973): “Information Theory and an Extention of the Maximum Likelihood

Principle” In B.N Petror and F. Csaki, ed. 2nd, International symposium on Inference Theory,

Akademia Kiado, Budapest, pp. 267-281.

[2] Al-Subaihi, Ali.A (2002): “Univariate variable selection Criteria available in SAS or SPSS”,

Paper presented at the ASA annual meeting, August 11-15, 2002, New York.

[3] Al-Subaihi, Ali.A (2007): “Variable Selection in Multivariate Regression using SAS / IML,

Saudi Arabia.

[4] Hannan, E.J and Quinn B.G (1979): “The Determination of the Order of an Autoregression”,

JRSS, B41, 190-195.

[5] Lutkepohl, H. (1985): “Comparison of Criteria for estimating the Order of a Vector

Autoregressive Process” Journal of Time Series Analysis vol. 6, 35 – 36, 47, 50.

[6] McQuarrie, A.D and Tsai (1998): “Regression and Time Series Model Selection”, World

Scientific Publishing Co. Ptc, LTD, River Edge, N.J

[7] Poskitt, D.S (1994). A note on autoregressive modeling, Econometric Theory, vol.10, 884-890

[8] Pötscher, B.M & Srinivasan, S. (1991). “A Comparison of Order estimation Procedures for

ARMA Models” Annals of Statistics 3, 20 – 21.

[9] Rissanen, J (1978):’ Modeling by Shotest data Description” Automatica, 14, pp. 445-471.

[10] Salau, M.O (2002): “On the accuracy and asymptotic convergence of widely used estimators

for autoregressive models”, Advances in Modeling and Analysis 2, 1-18.

[11] Shibata (1976): “Some Properties of the Order of an Autoregressive model selected by the

Generalization of Akaike’s EPF Criterion” Biometrika 1977 64(3):547-551;

doi:10.1093/biomet/64.3.547.

[12] Shibata (1976): “Selection of the Order of an Autoregressive Mode by Akaike Information

Criterion”, Biomertica, 63, pp.117-126.

[13] Shwartz, (1978): “Estimating the Dimension of a Model” Ann. Statist., 6, pp. 461-464.

[14] Stone, M (1979): “Comments on Model Selection Criteria of Akaike and Swartz” J.R.Statistics,

B41, No.2, pp. 276-278.