CHOOSING A QUALITY CRITERION FOR EVALUATING THE FORECAST OF NONLINEAR NON-STATIONARY PROCESSES
DOI:
https://doi.org/10.20535/kpisn.2021.2.236936Keywords:
mathematical modeling, forecasting, regression, forecast quality criteria, metrics, time series, nonstationary processesAbstract
Background. The problem of forecasting nonlinear nonstationary processes presented in the form of time series is very relevant, since such series can describe dynamics of the processes in both technical and economic systems. To establish the best model, various metrics are used to assess the quality of forecasts, such as R^2, RMSE, MAE, MAPE. However, in many tasks, when optimizing the model according to the selected criterion, the model becomes worse in relation to another criterion. Therefore it is important to understand which metric must be used to optimize and assess the quality of the forecast in the given task.
Objective. The aim of the paper is to develop a criteria base for assessing forecasts of nonlinear nonstationary processes, as well as an approach to choosing a metric in accordance to the specificity of the set forecasting problem.
Methods. The paper presents a comparative analysis of the basic metrics for the regression problem, their theoretical and practical meaning, advantages and disadvantages in various cases. New approaches are proposed based on the results of the analysis.
Results. Based on the analysis of the selected data, it is shown that by optimizing the model according to the selected criterion, the model becomes worse in relation to another criterion. A criterion basis for assessing forecasts of nonlinear nonstationary processes has been formed, as well as an approach to the selection of a quality criterion in accordance with the specifics of the set forecasting problem. To minimize an absolute error, the RMSE (MSE, R^2) and MAE metrics are analysed and recommended, depending on the need to work with outliers. The RMSLE metric is proposed for solving the problems of minimizing the relative metric, for solving the shown problems of the MAPE metric for this class of problems.
Conclusions. The paper shows the importance of choosing a metric that must be used to optimize and assess the quality of the forecasts in the given task. The obtained criterion base and approach can be used in further research to solve practical prob- lems in modelling and forecasting nonlinear nonstationary processes and to develop new methods or general method for solving such problems.
References
A.K. Palit and D. Popovic, Computational Intelligence in Time Series Forecasting: Theory and Engineering Applications. London, England: Springer-Verlag London, 2005, 372 p. doi: 10.1007/1-84628-184-9
O.M. Belas et al., “General approach to forecasting nonlinear nonstationary processes using mathematical models on statistical data,” Special Telecommun. Syst. Inform. Security, vol. 2, no. 4. pp. 26–31, 2018.
Walmart Recruiting – Store Sales Forecasting [Online]. Avaliable: https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting/data.
O.M. Belas et al., “Comparative analysis of autoregressive approaches and recurrent neural networks for modeling and forecasting of nonlinear nonstationary processes,” Inform. Technol. Security, vol. 7, no. 1, pp. 91–99, 2019.
P.I. Bidyuk et al., Time Series Analysis. Kyiv, Ukraine: Polytechnica, 2010, 317 p.
R.J. Hyndman and G. Athanasopoulos, Forecasting: Principles and Practice. Melbourne, Australia: OTexts, 2013.
I. Goodfellow et al., Deep Learning. New York: MIT Press, 2016, 800 p.
C. Bishop, Pattern Recognition and Machine Learning. New York: Springer-Verlag New York, 2006, 738 p.
S. Kolassa, “Evaluating predictive count data distributions in retail sales forecasting,” Int. J. Forecasting, vol. 32, no. 3, pp. 788–803, 2016. doi: 10.1016/j.ijforecast.2015.12.004
E.F. Krause, Taxicab Geometry: An Adventure in Non-Euclidean Geometry. New York: Dover Publications, 1987, 96 p.
J. S. Armstrong, Long-range Forecasting: From Crystal Ball to Computer. New York: John Wiley & Sons, 1978.
S. Makridakis, “Accuracy measures: theoretical and practical concerns,” Int. J. Forecasting, vol. 9, no. 4, pp. 527–529, 1993. doi: 10.1016/0169-2070(93)90079-3
P. Goodwin and R. Lawton, “On the asymmetry of the symmetric MAPE,” Int. J. Forecasting, vol. 15, no. 4, pp. 405–408, 1999. doi: 10.1016/S0169-2070(99)00007-2
R.J. Hyndman and A.B. Koehler, “Another look at measures of forecast accuracy,” Int. J. Forecasting, vol. 22, no. 4, pp. 679–688, 2006. doi: 10.1016/j.ijforecast.2006.03.001
F. Petropoulos and N. Kourentzes, “Forecast combinations for intermittent demand,” J. Operat. Res. Soc., vol. 66, no. 6, pp. 914–924, 2015, doi: 10.1057/jors.2014.62
A. Davydenko and R. Fildes, “Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts,” Int. J. Forecasting, vol. 29, no. 3, pp. 510–522, 2013. doi: 10.1016/j.ijforecast.2012.09.002
Downloads
Published
Issue
Section
License
Copyright (c) 2021 Andrii O. Belas, Petro I. Bidyuk
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under CC BY 4.0 that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work
