MethodPDF Available

Abstract

A checklist of evidence-based standards for forecasting. The checklist is a version of one that appeared in Armstrong's (2001) "Principles of Forecasting: A Handbook for Researchers and Practitioners"
FORECASTING STANDARDS CHECKLIST
An electronic version of this checklist is available on the Forecasting Principles Web site.
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
PROBLEM
1. Setting Objectives
1.1. Describe decisions that might be affected
1.2. Agree on actions for different possible
forecasts
1.3. Make forecast independent of organizational
politics
1.4. Consider whether events or series are
forecastable
1.5. Gain decision makers’ agreement on methods
2. Structuring the Problem
2.1. Identify possible outcomes prior to making
forecasts
2.2. Tailor the level of data aggregation to the
decisions
2.3. Decompose the problem into sub problems
2.4. Decompose time series by causal forces
2.5. Structure problems to deal with important
interactions
2.6. Structure problems that involve causal chains
2.7. Decompose time series by level and trend
INFORMATION
3. Identifying Information Sources
3.1. Use theory to guide information search on
explanatory variables
3.2. Ensure that data match the forecasting
situation
3.3. Avoid biased data sources
3.4. Use diverse sources of data
3.5. Obtain information from similar (analogous)
series or cases
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
4. Collecting Data
4.1. Use unbiased and systematic procedures to
collect data
4.2. Ensure that information is reliable
4.3. Ensure information is valid
4.4. Obtain all important data
4.5. Avoid collection of irrelevant data
4.6. Obtain the most recent data
5. Preparing Data
5.1. Clean the data
5.2. Use transformations as required by
expectations
5.3. Adjust intermittent series
5.4. Adjust for unsystematic past events (outliers)
5.5. Adjust for systematic events (e.g., seasonality)
5.6. Use multiplicative adjustments for seasonality
for stable series with trends
5.7. Damp seasonal factors for uncertainty
5.8. Use graphical displays for data
METHODS
6. Selecting Methods
6.1. Develop list of all important criteria
6.2. Ask unbiased experts to rate potential methods
6.3. Use structured forecasting methods rather than
unstructured
6.4. Use quantitative methods rather than
qualitative methods
6.5. Use causal rather than naïve methods
6.6. Select simple methods unless evidence favors
complex methods
6.7 Match forecasting method(s) to the situation
6.8. Compare track records of various methods
6.9. Assess acceptability and understandability of
methods to users
6.10. Examine value of alternative forecasting
methods
Forecasting Standards Checklist 3
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
7. Implementing Methods: General
7.1. Keep methods simple
7.2. Provide a realistic representation of the
forecasting situation
7.3. Be conservative in situations of uncertainty or
instability
7.4. Do not forecast cycles
7.5. Adjust for expected events in future
7.6. Pool similar types of data
7.7. Ensure consistency with forecasts of related
series
8. Implementing Methods: Judgment
8.1. Pretest questions used to solicit judgmental
forecasts
8.2. Use questions that have been framed in
alternative ways
8.3. Ask experts to justify their forecasts
8.4. Use numerical scales with several categories
8.5. Obtain forecasts from heterogeneous experts
8.6. Obtain intentions or expectations from
representative samples
8.7. Obtain forecasts from sufficient number of
respondents
8.8. Obtain multiple estimates of an event from
each expert
9. Implementing Method: Quantitative
9.1. Tailor the forecasting model to the horizon
9.2. Match model to underlying process
9.3. Do not use fit to develop a model
9.4. Weight the most relevant data more heavily
9.5. Update models frequently
10. Implementing Methods: Quantitative Models
with Explanatory Variables
10.1. Use theory and domain expertise to select
casual variables
10.2. Use all important variables
10.3. Use theory and domain expertise to specify
directions of relationships
10.4. Use theory and domain expertise to
estimate/limit relationships
10.5. Use different types of data to estimate a
relationship
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
10.6. Forecast for at least two alternative
environments
10.7. Forecast for alternative interventions
10.8. Apply the same principles to the forecasts of
the explanatory variables
10.9. Shrink the forecasts of change if there is
uncertainty for predictions of the explanatory
variables
11. Integrating Judgmental and Quantitative
Methods
11.1. Use structured procedures to do the integration
11.2. Use structured judgment as inputs to models
11.3. Use prespecified domain knowledge as input
in selecting, weighting, and modifying
quantitative methods
11.4. Limit subjective adjustments of quantitative
forecasts
11.5. Use judgmental bootstrapping instead of
expert forecasts
12. Combining Forecasts
12.1. Combine forecasts from approaches that differ
12.2. Use many approaches (or forecasters),
preferably at least five
12.3. Use formal procedures to combine forecasts
12.4. Start with equal weights
12.5. Use trimmed means
12.6. Use evidence on each method’s accuracy to
vary the weights on the component forecasts.
12.7. Use domain knowledge to vary the weights on
the component forecasts
12.8. Combine when there is uncertainty about
which method is best
12.9. Combine when uncertainty exists about the
situation
12.10. Combine when it is important to avoid large
errors
Forecasting Standards Checklist 5
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
EVALUATION
13. Evaluating Methods
13.1. Compare reasonable methods
13.2. Use objective tests of assumptions
13.3. Design test situation to match the forecasting
problem
13.4. Describe conditions associated with the
forecasting problem
13.5. Tailor the analysis to the decision
13.6. Describe potential forecaster biases
13.7. Assess reliability and validity of the data
13.8. Provide easy access to the data
13.9. Provide full disclosure of methods
13.10. Test assumptions for validity
13.11. Test client’s understanding of the methods
13.12. Use direct replications of the evaluations to
identify mistakes
13.13. Use replications of the forecast evaluations to
assess reliability
13.14. Use extensions of evaluations for
generalizability
13.15. Conduct extensions of evaluations in realistic
situations
13.16. Compare forecasts generated by different
methods
13.17. Examine all important criteria
13.18. Specify criteria prior to analyzing the data
13.19. Assess face validity
13.20. Use error measures that adjust for scale
13.21. Ensure error measures are valid
13.22. Use error measures that are not sensitive to
degree of difficulty in forecasting
13.23. Avoid biased error measure
13.24. Avoid error measures with high sensitivity to
outliers
13.25. Use multiple measures of accuracy
13.26. Use out-of-sample (ex ante) error measures
13.27. Use ex post accuracy test to evaluate effects
13.28. Do not use adjusted R-square to compare
models
13.29. Use statistical significance only to compare
the accuracy of reasonable methods
13.30. Do not use root-mean-square errors to make
comparisons
Does formal procedure follow the
standard?
NO! YES!
N/A –2 –1 0 1 2 ?
13.31. Base comparisons on large sample
13.32. Conduct explicit cost-benefit analyses
14. Assessing Uncertainty
14.1. Estimate prediction intervals (PI)
14.2. Use objective procedures
14.3. Develop PI using realistic representation of
the situation
14.4. Use transformations when needed to estimate
symmetric PIs
14.5. Ensure consistency over forecast horizon
14.6. List reasons why forecast might be wrong
14.7. Consider likelihood of alternative outcomes
in assessing PIs
14.8. Obtain good feedback on accuracy and
reasons for errors
14.9. Combine PIs from alternative methods
14.10. Use safety factors for PIs
14.11. Conduct experiments
14.12. Do not assess uncertainty in a traditional
group meeting
14.13. Incorporate the uncertainty for predictions of
the explanatory variables
14.14. Ask for a judgmental likelihood that a
forecast will fall within a pre-defined
minimum-maximum interval
USING FORECASTS
15. Presenting Forecasts
15.1. Provide clear summary of forecasts and data
15.2. Provide clear explanation of methods
15.3. Describe assumptions
15.4. Present prediction intervals
15.5. Present forecasts as scenarios
16. Learning
16.1. Consider use of adaptive models
16.2. Seek feedback about forecasts
16.3. Use a formal review process for forecasting
methods
16.4. Use a formal review process for use of
forecasts
ResearchGate has not been able to resolve any citations for this publication.
ResearchGate has not been able to resolve any references for this publication.