Content uploaded by Olanrewaju Ismail Shittu
Author content
All content in this area was uploaded by Olanrewaju Ismail Shittu on Apr 07, 2014
Content may be subject to copyright.
European Journal of Scientific Research
ISSN 1450-216X Vol.30 No.3 (2009), pp.409-416
© EuroJournals Publishing, Inc. 2009
http://www.eurojournals.com/ejsr.htm
Comparison of Criteria for Estimating the Order of
Autoregressive Process: A Monte Carlo Approach
Shittu. O.I
Department of Statistics, University of Ibadan, Ibadan, Nigeria
E-mail: shiittu.olanrewaju@gmail.com
Asemota. M.J
Department of Statistics, University of Ibadan, Ibadan, Nigeria
Abstract
This paper compares the performance of model order determination criteria in terms
of selecting the correct order of an Autoregressive model in small and large samples using
the simulation method. The considered criteria are the Akaike information criterion (AIC);
Bayesian information criterion (BIC) and the Hannan − Quinn criterion (HQ). Eight series
of sizes N=50 and N=200 were generated with 50 replicates each under consistent
assumptions and parameter specifications.
Our results shows that Bayesian information criteria (BIC) performs best in terms of
selecting the correct order of an Autoregressive model for small samples irrespective of the
AR structure, Hannan Quinn criteria can be said to perform best in large sample. Even
though the AIC has the least performance among the criteria considered, it appears to be
the best in terms of the closeness of the selected order to the true value. Our results
compares favorably with that of Lutkepohl (1985) and Benedikt Pötscher (1991).
Keywords: Monte Carlo; Information, Criteria; Performance; Simulation.
1. Introduction
It is well known that most economic and financial series follow the Autoregressive Moving average
[ARMA(p,q)] model, and more often than not the autoregressive [AR(p)] model where p and/or q are
the order of the model. However, determination of the correct order p,q has been a source of serious
concern to analysts over the years, since inappropriate order selection may result into inconsistent
estimate of parameters if p< true value or may not be consistent and increase in the variance of the
model if p>true, (Shibata, 1976).
In recent years there has been a substantial literature on this problem and different criteria have
been proposed to aid in choosing the order of the ARMA(p,q) process. These criteria are based on
theoretical considerations that provide only asymptotic properties of the resulting estimators. The
practitioner, however, usually faces the problem of making a choice on the basis of a limited data set.
Among the criteria considered are the Akaike information criteria (AIC); Bayesian Information
criteria (BIC), Hannan Quinne criteria (HQ); Carlos Information criteria (CIC) and others.
This paper therefore compares the performance of the commonly used information criteria such
as: Akaike Information Criteria (AIC), Bayesian Information Criteria (BIC), and Hannan – Quinn
Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte
Carlo Approach 410
(HQ) Criterion, for the Autoregressive AR(p) model using the Monte Carlo simulation method for
small and large samples.
2. Review of Literature
The information criterion has been widely used in time series analysis to determine the appropriate
order of a model. The information criteria are often used as a guide in model selection. The goal of any
order selection procedure is to estimate the order p for an AR model on the basis of n successive
observations from a time series X(t) while the notion of an information criterion is to provide a
measure of information in terms of the order of the model, that strikes a balance between these
measures of goodness of fit and parsimonious specification of the model. The Kullback – Leibler
quantity of information contained in a model is the distance from the “true” model and is measured by
the log likelihood function.
Several criteria are used for this purpose, in particular, we discuss the AIC (Akaike, 1974), BIC
(Rissanen, 1978; Scwarz, 1978) and HQ (Hannan and Quinn, 1979) amongst others.
All these criteria aim at minimizing the residual sum of squares and impose a penalty for
including an increasingly large number of regressors (lag values).
2.1. Akaike Information Criterion
The Akaike Information Criterion (AIC), (Akaike, 1974) is an objective measure of model suitability
which balances model fit and model complexity. Considering a stationary time series {Xt}, t = 1, 2, …
N, the Akaike information criteria consist of minimizing the function:
K(p) = N loge2
p
σ
+ pc(n) (3.1)
p∈P = { 0, 1, 2, … m}
where 2
p
σ
is the estimated residual variance for a fitted AR (p) model, c(n) is a penalty term, N is the
number of observations and m is a pre – determined upper autoregressive order.
To obtain
AIC (p) = 2
log pe
N
σ
+ 2p (3.2)
p ∈ P= { 0, 1, 2, … m}
where
2
11
21∑∑
==
−⎥
⎦
⎤
⎢
⎣
⎡−= N
t
p
jjtjtp XX
N
φσ
or more compactly as; ∑
=22 1tp N
εσ
selection of the chosen model is then made by considering the minimum
AIC = min {AIC(p)}, that is the model with smallest AIC is chosen.
Details of the proof can be found in Akaike (1974) and Shibata (1976).
One advantage of AIC is that it is useful for not only in – sample data but also out–of–sample
forecasting performance of a model. In – sample forecasting essentially tells us how the chosen model
fits the data in a given sample while the out–of–sample forecasting is concerned with determining how
a fitted model forecasts future values of the regressed, given the values of the repressors. It is also
useful for both nested and non-nested models. The outstanding demerit of this criterion is that of
inconsistency. The procedure has been criticized because it is inconsistent and tends to over– fit a
model, Shibata(1976) showed this for autoregressive model and Hannan(1982) for ARMA models.
411 Shittu. O.I and Asemota. M.J
2.2. Bayesian Information Criteria
The Bayesian information criterion is obtained by replacing the non-negative function c(n) in (2.2) by
loge(N). Hence, we have
BIC(p) = NpN epe loglog 2+
∧
σ
(3.3)
p ∈ P = {0, 1, 2, … m}
where ∧2
p
σ
is obtained as above and the appropriate model is obtained as that which minimizes the BIC
(p) above, that is, min (BIC(p))
Details of the discussion can be found in Rissanen (1978); Schwarz (1978) and Stone (1979).
The BIC imposes a harsher penalty than AIC, as its obvious from comparing (3.2) to (3.3). An
important advantage of BIC is that for a wide range of statistical problems, it is order consistent (i.e.
when the sample size grows to infinity, the probability of choosing the right model converges to unity)
leading to more parsimonious models. Like the AIC, the lower the value of BIC, the better the model.
Like AIC, BIC can be used to compare in–sample or out–of–sample forecasting performance of a
model.
2.3. Hannan – Quinn Criterion
The Hannan-Quinn criterion for identifying an autoregressive model denoted by HQ (p) was
introduced by Hannan and Quinn (1979). The adjusted version of it can also be applied to regression
models, Al-Subaihi (2007). It is obtained by replacing the non – negative penalty function c(n) in
equation (3.2) by ClogClogeN.
Thus, we have
HQ(p) = Nloge ∧2
p
σ
+ p C ln Cloge(N) (3.4)
p ∈ P = {0, 1, … m}.
where C is a constant, C > 2. For practical purpose, we set C = 2. The best model is the model that
correspond to minimum HQ i.e min (HQ (p))
The order selection procedure presented above have the advantage of being objective and
automatic, but it over-fit when the sample size is small.
Detailed discussion on this can be found in Hannan-Quinn (1979); McQuarrie and Tsai (1998).
Having noticed the limitations of these criteria, the aim of this study is to examine the veracity
of these claims and compare their performance using the Monte Carlo study.
3. Simulation Study
The basic theory behind random number generation with computers offers a simple example of Monte
Carlo simulation to understand the properties of different statistics computed from sample data. Monte
Carlo methods comprise that branch of experimental mathematics, which is concerned with utilization
of random normal deviates. The random deviates are generated using Microsoft Excel random number
generator to have zero mean and unit standard deviation, N(0,1). The routine generates realization for a
given AR structure.
In order to contrast the performance of the order identification criteria, simulation study was
conducted using a wide range of autoregressive (AR) processes with different characteristics.
The following assumptions were made. The random numbers follow a standard normal
distribution with mean zero and variance unity. Data were generated for samples of N = 50 and N =
200 with 50 replications each. Autoregressive models of orders p = 0, 1, 2, 3, and 4 were generated
with a maximum of 5. The parameters of all the generated series in the simulation study are chosen so
that the series are stationary that is the roots fall outside the unit circle. The autoregressive structures
p1 to p8 and their parameters are given in the table below:
Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte
Carlo Approach 412
Table 1: Data Generating Processes
Process AR Structure Parameter Values
AR (1)
P1 ROOT: 10 φ = 0.1
AR (1)
P2 ROOT: - 1.11 φ = - 0.9
AR (2) φ1 = 0.7
P3 ROOT: 5, 2 φ2 = - 0.1
AR (2) φ1 = 0.8
P4 ROOT: - 1.45, - 9.22 φ2 = 0.075
AR (3) φ1 = 0.8, φ2 = 0.3
P5 ROOT: 2.29, 1.13, - 1.93 φ3 = - 0.2
AR (3) φ1 = 0.9, φ2 = 0.5
P6 ROOT: - 1.30, 1.07+I, 1.07 – i φ3 = - 0.6
AR (4) φ1 = φ2 = φ3 = 0
P7 ROOT: ± 1.19 four-times φ4 = 0.5
AR (4) φ1 = 0.9, φ2 = 0.7
P8 ROOT: 1.11, 1.142, 2.00, 3.33 φ3 = 0.5, φ4 = 0.3
From Table 1 above, we have two processes for each of the structures AR(1), AR(1), AR(3)
and AR (4). Process p1 is an autoregressive structure of order (1) having its root located far away from
the unit circle, while process p2 is an AR (1) structure with its root close to the unit circle. Process p3 is
an AR (2) structure with the two roots located further away from the unit circle, process p4 is also AR
(2) structure, with one of its roots very close to the unit circle.
Process p5 is an AR (3) structure with two of its roots very close to the unit circle, while process
p6 is also an AR (3) structure wit two of its roots being complex while the other falls barely outside the
unit circle. This scenario represents a situation of a mixture of real and complex roots.
Process p7 is an AR(4) structure with repeated/multiple roots, while process p8 is also an AR (4)
structure which has two of its roots close to the unit circle while the other two roots are some distance
from the unit circle.
For each of the model structure, an autoregressive model was fit the model order were
examined using the criteria AIC, BIC and HQ using the E – views software package.
The performance criterion is that, the information criterion with the highest number of cases (or
percentage) of selecting the correct order of the given AR structure is considered to be the best.
4. Results and Discussions
After fitting the models to the generated data, the number of times each identification criteria (AIC,
BIC and HQ) was able to accurately identity the correct order of a given AR structure were counted
and frequencies obtained.
In measuring the performance of the generated models, the criterion that has the highest
number of cases (or percentage frequency) of selecting the correct order of the given AR structure is
considered to be the best criterion. The results of the analysis were tabulated different criterion for N =
50 in able 2 and for N = 200 in Table 3.
413 Shittu. O.I and Asemota. M.J
Table 2: Frequency And Percentage Distribution Of From Simulated Data When N = 50 And R = 50
Order Selected
AR Processes Criterion 0 1 2 3 4 5
AIC -
40(80%) 8(16%) 2(4%) - -
BIC -
45(90%) 4(8%) 1(2%) - -
P1 HQ -
42(84%) 6(12%) 2(4%) - -
AIC 1(2%) 36(72%) 10(20%) 3(6%) - -
BIC -
43(86%) 5(10%) 2(4%) - -
P2 HQ 1(2%) 40(80%) 8(16%) 1(2%) - -
AIC 2(4%) 8(16%) 35(70%) 5(10%) - -
BIC 1(2%) 7(14%) 36(72%) 6(12%) - -
P3 HQ -
13(26%) 30(60%) 7(14%) - -
AIC 1(2%) 2(4%) 43(86%) 4(8%) - -
BIC 1(2%) 3(6%) 44(88%) 2(4%) - -
P4 HQ 1(2%) 3(6%) 42(84%) 4(8%) - -
AIC -
2(4%) 4(8%) 38(76%) 6(12%) -
BIC - -
6(12%) 41(82%) 3(6%) -
P5 HQ - -
7(14%) 42(84%) 1(2%) -
AIC -
1(2%) 5(10%) 40(80%) 4(8%) -
BIC - -
3(6%) 46(92%) 1(2%) -
P6 HQ - -
4(8%) 44(88%) 2(4%) -
AIC - -
3(6%) 4(8%) 40(80%) 3(6%)
BIC - - -
1(2%) 48(96%) 1(2%)
P7 HQ - - -
2(4%) 47(94%) 1(2%)
AIC - - -
4(8%) 44(88%) 2(4%)
BIC - - -
3(6%) 46(92%) 1(2%)
P8 HQ - -
1(2%) 6(12%) 41(82%) 2(4%)
From the above table for small sample (N = 50), it can be observed that BIC criterion performs
best in terms of the percentage number of correct order identified in various AR models regardless of
the parameterization and sample size. This is closely followed by Hannan-Quinn criterion.
Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte
Carlo Approach 414
Table 3: Frequency And Percentage Distribution Of Order From Simulated Data When N = 200 And R = 50
Order Selected
AR Processes Criterion 0 1 2 3 4 5
AIC 1(2%) 40(80%) 5(10%) 4(8%) - -
BIC -
44(88%) 3(6%) 3(6%) - -
P1 HQ 1(2%) 46(92%) 2(4%) 1(2%) - -
AIC 2(4%) 40(80%) 6(12%) 2(4%) - -
BIC -
45(90%) 3(6%) 2(4%) - -
P2 HQ 1(2%) 43(86%) 5(10%) 1(2%) - -
AIC 1(2%) 8(16%) 38(76%) 3(6%) - -
BIC -
3(6%) 46(92%) 1(2%) - -
P3 HQ -
3(6%) 42(84%) 5(10%) - -
AIC -
7(14%) 42(84%) 1(2%) - -
BIC -
5(10%) 45(90%) - - -
P4 HQ -
2(4%) 47(94%) 1(2%) - -
AIC -
1(2%) 4(8%) 41(82%) 4(8%) -
BIC - -
1(2%) 48(96%) 1(2%) -
P5 HQ - -
2(4%) 45(90%) 3(6%) -
AIC -
1(2%) 3(6%) 40(80%) 6(12%) -
BIC -
1(2%) 3(6%) 44(88%) 2(4%) -
P6 HQ -
2(4%) 4(8%) 43(86%) 1(2%) -
AIC - -
2(4%) 4(8%) 41(82%) 3(6%)
BIC -
1(2%) 2(4%) 2(4%) 42(84%) 3(6%)
P7 HQ - - -
2(4%) 45(90%) 3(6%)
AIC - - -
6(12%) 40(80%) 4(8%)
BIC - - -
1(2%) 47(94%) 2(4%)
P8 HQ - - -
2(4%) 45(90%) 3(6%)
From Table 3, we found that the BIC performed better in five out of the eight AR processes
considered. The HQ followed closely with even a better performance in three of the AR process.
The AIC performance was the least in all the processes considered. From these results, it
appears that the performance of HQ criterion is slightly improved with increase in sample size, this is
not surprising in the sense that, in principle, as the sample size increased, convergence is expected to
occur for all criteria especially for Hannan-Quinn criterion. however both the AIC and HQ exhibit
highest rate of inconsistency in their estimation as they have higher rate of over estimation or under
estimation of the order of the models. We also found that both the AIC and HQ exhibit highest rate of
inconsistency in their estimation as they have higher rate of over estimation or under estimation of the
order of the models. This result agrees with that of Poskitt (1994) and Salau (2002).
The closeness of the value of the selected order to the true value for each of the information
criteria is given in the table below.
Table 4: Average Number Of Ar Order (P) Selected For Simulated Data
Model P1 P = 1 P2 P = 1 P3 P = 2 P4 P = 2 P5 P = 3 P6 P = 3 P7 P = 4 P8 P = 4
N 50 200 50 200 50 200 50 200 50 200 50 200 50 200 50 200
Observed Average
value of p 1.1 1.2 1.1 1.3 1.8 1.9 1.8 1.9 2.9 3.0 2.9 3.0 3.9 4.0 3.9 4.0
AIC 4 4 0 6 6 6 4 8 6 6 4 2 6 0 8 6
BIC 2 8 8 4 4 6 4 0 4 0 6 2 8 0 6 2
HQ 6 2 8 2 8 4 8 8 8 2 6 6 8 2 8 2
415 Shittu. O.I and Asemota. M.J
Table 4 displays the average order selected by the criteria as compared with the specified ‘true’
orderHence on the average, the BIC is seen to outperform both the HQ and AIC, regardless of the
sample size and the AR structures considered, with the AIC having the least performance.
5. Summary and Conclusion
Based on the result of our simulation study, we conclude that the Bayesian information criteria (BIC)
performs best in terms of selecting the correct order of an Autoregressive model for small samples
irrespective of the AR structure, parameter values and nature of the roots of the resulting polynomial.
The Hannan Quinn criteria can be said to perform best in large sample as it performed best in
five of the eight cases considered for large samples and its capacity for estimating the true order the
next best on the remaining cases. The performance of AIC improved with increase in sample size but
still had the poorest performance in all the eight processes. It can also be observed that both AIC and
HQ were mostly inconsistent in their estimation for both small and large samples.
It was also found that orders selected by HQ were closes to the true value for small sample size
while BIC was closest for large samples. Even though the AIC has the least performance among the
criteria considered, it appears to be the best in terms of the closeness of the selected order to the true
value. These research findings are quite in agreement with the work of Lutkepohl (1985) for vector
autoregressive process (VAR) and the works of Benedikt Pötscher (1991) on ARMA models.
However, according to Al-Subaihi(2002), it is important for an investigator not to depend
entirely on an information criterion in evaluating the order of a model because non of them works well
under all conditions. Their performance depends on sample size, number of dependent and independent
variable as well as the correlation between them in the case of multiple regression models.
A researcher needs to evaluate the ‘good’ model using various model order criteria bearing in
mind the conditions mentioned above.
Acknowledgement
This paper is partly funded by Senate research grant number SRG/FSC/2006/2A of the University of
Ibadan, Nigeria.
Comparison of Criteria for Estimating the Order of Autoregressive Process: A Monte
Carlo Approach 416
References
[1] Akaike, H (1973): “Information Theory and an Extention of the Maximum Likelihood
Principle” In B.N Petror and F. Csaki, ed. 2nd, International symposium on Inference Theory,
Akademia Kiado, Budapest, pp. 267-281.
[2] Al-Subaihi, Ali.A (2002): “Univariate variable selection Criteria available in SAS or SPSS”,
Paper presented at the ASA annual meeting, August 11-15, 2002, New York.
[3] Al-Subaihi, Ali.A (2007): “Variable Selection in Multivariate Regression using SAS / IML,
Saudi Arabia.
[4] Hannan, E.J and Quinn B.G (1979): “The Determination of the Order of an Autoregression”,
JRSS, B41, 190-195.
[5] Lutkepohl, H. (1985): “Comparison of Criteria for estimating the Order of a Vector
Autoregressive Process” Journal of Time Series Analysis vol. 6, 35 – 36, 47, 50.
[6] McQuarrie, A.D and Tsai (1998): “Regression and Time Series Model Selection”, World
Scientific Publishing Co. Ptc, LTD, River Edge, N.J
[7] Poskitt, D.S (1994). A note on autoregressive modeling, Econometric Theory, vol.10, 884-890
[8] Pötscher, B.M & Srinivasan, S. (1991). “A Comparison of Order estimation Procedures for
ARMA Models” Annals of Statistics 3, 20 – 21.
[9] Rissanen, J (1978):’ Modeling by Shotest data Description” Automatica, 14, pp. 445-471.
[10] Salau, M.O (2002): “On the accuracy and asymptotic convergence of widely used estimators
for autoregressive models”, Advances in Modeling and Analysis 2, 1-18.
[11] Shibata (1976): “Some Properties of the Order of an Autoregressive model selected by the
Generalization of Akaike’s EPF Criterion” Biometrika 1977 64(3):547-551;
doi:10.1093/biomet/64.3.547.
[12] Shibata (1976): “Selection of the Order of an Autoregressive Mode by Akaike Information
Criterion”, Biomertica, 63, pp.117-126.
[13] Shwartz, (1978): “Estimating the Dimension of a Model” Ann. Statist., 6, pp. 461-464.
[14] Stone, M (1979): “Comments on Model Selection Criteria of Akaike and Swartz” J.R.Statistics,
B41, No.2, pp. 276-278.
