Adv Syst Sci Appl 2023; 04:8–17 Published online at https://ijassa.ipu.ru. A State Space Filtering-Based Approach for Price Prediction Anton Belyakov12, Aleksei Kurbatskii1*, Artur Sidorenko1 1Moscow School of Economics, Lomonosov Moscow State University, Moscow, Russia 2 Moscow Institute of Steel and Alloys ‘‘MISiS’’, Moscow, Russia Abstract: We present a method of the forecasting and the data filtering of a linear dynamic system based on the dimension reduction of the space of unobservable states. The method relies on the singular value decomposition of the Hankel matrix. The decomposition is used to calculate unknown parameters of the model. The elements of the singular value decomposition are separated into blocks enabling to estimate the initial state and the system matrices and predict the system dynamics and the data filtering by identifying exponential trends and periods of seasonal fluctuations. To illustrate the quality of fitting and the determined periods of an oscillatory system with trends and the white noise, we conducted numerical simulations of such systems. The parameter estimates were obtained with high precision. Then, daily electricity price data from the NordPool system from 2016 to 2020 were used to generate in-sample and out-of-sample forecasts. The advantages of the proposed method include the ability to handle ill-conditioned matrices and to determine the periods of oscillatory systems. This is significant due to the presence of seasonality in many economic indicators. In the analyzed daily electricity price data, the method identified the presence of biweekly and monthly seasonality. Keywords: state space method, principal components, time series, singular value decomposi- tion, filter, electricity price forecasting 1. INTRODUCTION The representation of systems in the state space is a common way to describe the dynamics of linear systems. State space models constitute one of the classes of time series models that are successfully applied in management of technical systems and solving system identification problems, as well as in the fields of filtering and forecasting in economics and finance. Descriptions of these models in economic and financial contexts can be found in [13,15,17]. These works also discuss parameter estimation methods for such models, including maximum likelihood and Bayesian methods. One of the first approaches in this paradigm is the renowned Kalman filter, see, e.g., [12]. Among the earliest applications of state space methods and the Kalman filter, particularly in economics and finance, notable works include [5, 8]. In this work, in addition to filtering, the problem of estimating the hyperparameters of the state space model is considered. In such situations, making forecasts includes preliminary estimation of unknown hyperparameters. In this regard, we rely on and develop the methodology described in [2, 16]. The model used in this work is as follows: there is a linear evolution of hidden states, where observed variables are obtained by projecting from the space of hidden unobservable ∗Corresponding author: akurbatskiy@gmail.com A STATE SPACE FILTERING-BASED APPROACH FOR PRICE PREDICTION 9 variables. Algebraically, this concept is expressed as a system of two vector linear equations, where one difference equation describes the change in the hidden state xt over time t, while the other relates the observed parameters yt and the unobservable parameters xt. Random factors influence the system, causing the proposed dynamics not to be fully reproduced. Random deviations are modeled as mutually independent random variables. To identify the system, it is necessary to estimate the dimension of the vector xt of hidden variables, the initial state x1, and the corresponding representation matrices in state space. The dimension of the hidden state space is a hyperparameter that is either externally specified or determined by cross-validation methods. The approach based on the singular value decomposition of the Hankel matrix is used to calculate the unknown parameters of the model [2, 3, 10, 16]. This study focuses on forecasting daily electricity prices from 2016 to 2020, and additionally proposes a data filtering method. On a sample of 1170 observations, it was found that the short-term out-of-sample forecast is comparable to traditional econometric models and to the well-known Holt–Winters’ filtering method based on the mean absolute error indicator even in a stationary segment of the time series. In addition to modeling on real data and comparing the forecast quality of the proposed method with well-known models, we present several numerical simulations. These simulations demonstrate the specific characteristics of data behavior that the proposed state space model can identify. 2. BASICS In general, an autoregressive system can be represented in the state space as follows: xt+1 = Axt + ξt, yt = Cxt + ηt, where t ∈ N, xt is an unobservable k × 1 state vector, x1 is deterministic (but unobservable), yt is an observable l × 1 vector, (ξt, ηt)t∈N is an unobservable white noise, and A and C are state space matrices. It is easily seen that the expected value Eyt+1 = CAtx1. For the sake of simplicity, we set ξt = 0 and ηt = 0 for all t. In such a case, yt+1 = CAtx1. (2.1) Having a sample of size T the Hankel matrix H can be written as follows for an odd-sized sample: H =  y1 y2 . . . yn y2 y3 . . . yn+1 ... ... . . . ... yn yn+1 . . . y2n−1  = Γ1:nΩ1:n (2.2) and for an even-sized sample: H =  y1 y2 . . . yn+1 y2 y3 . . . yn+2 ... ... . . . ... yn yn+1 . . . y2n.  = Γ1:nΩ1:(n+1), (2.3) Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 10 A. BELYAKOV, A. KURBATSKII, A. SIDORENKO where n = [ T 2 ] . By virtue of formula (2.1) the Hankel matrix is the product of the observability matrix Γ1:n =  C CA ... CAn−1  and the controllability matrix Ω1:n = [x1 Ax1 . . . An−1x1] for an odd-sized sample, or Ω1:(n+1) = [x1 Ax1 . . . Anx1] for an even-sized sample. Therefore, if we find a decomposition of the Hankel matrix as a product of two matrices as in (2.2) or (2.3), we can estimate the matrices of the system: Ĉ = Γ̂1:1,  = ( Γ̂1:(n−1) )+ Γ̂2:n, x̂1 = Ω̂1:1, (2.4) where ( Γ̂1:(n−1) )+ is the Moore–Penrose pseudoinverse of the matrix Γ̂1:(n−1) excluding the last block, and the matrix Γ̂2:n does not include the first block. The hat sign indicates that these are estimates of matrices Γ and Ω that will be suggested now. As in [16], we apply the singular value decomposition of the observed matrix Ĥ = Û ŜV̂ ′, where Ŝ is a diagonal matrix of singular values arranged in decreasing order, and Û and V̂ are orthogonal matrices. We consider an odd T since the case for even T is similar. These matrices can be split into parts corresponding to, as we assume, the signal and noise: Û = [ Û1:k Û(k+1):n ] , V̂ = [ V̂1:k V̂(k+1):n ] , Ŝ = [ Ŝ1:k 0 0 Ŝ(k+1):n ] , where Û1:k and V̂1:k are the matrices consisting of first k columns of matrices Û and V̂ correspondingly, while Ŝ1:k is the square k × k matrix having in its diagonal first k singular values of Ĥ . The estimates for the observability and controllability matrices can be found as follows: Γ̂1:n = Û1:kŜ 1/2 1:k , Ω̂1:n = Ŝ 1/2 1:k V̂ ′ 1:k. Once the estimates Ĉ,  and x̂1 are obtained from (2.4), formula (2.1) can be used for forecasting. It should be noted that matrices C, A and vector x1 are determined, in general, up to a non-degenerate linear transformation with matrix P : C̃ = CP−1, à = PAP−1, x̃1 = Px1. But, for forecasting and filtering, it doesn’t matter. However, in practice, formula (2.1) can lead to significant computational errors, as the matrix A is multiplied by itself many times. To find a more accurate estimate, let us express (2.1) as yt = CAn−1At−TAn−1x1 = Γn :nA t−TΩn :n, considering odd T . A suggested estimate for t ≥ T is the following ŷt = Γ̂n :n t−T Ω̂n :n. In addition to the forecasting task, we propose a possible solution to the data filtering problem. Let’s consider the case for odd T , as for even T the calculations are similar. It is Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) A STATE SPACE FILTERING-BASED APPROACH FOR PRICE PREDICTION 11 noted that for t ≤ T yt = Γk : kΩ(t−k+1) : (t−k+1), where 1 ≤ k ≤ t. The proposed estimate is set as an average value of estimates with different parameters k in the following way ŷt = 1 min(t, n)−max(0, t+ 1− n) min(t,n)∑ k=max(0,t+1−n) Γ̂k : kΩ̂(t−k+1) : (t−k+1). In order to select the dimension k of the state vector (State Space Dimension, SSD), one may apply any cross-validation method, e.g. the Mean Absolute Error (MAE) of the one-day- ahead (or n-days-ahead) forecast in a rolling-sample scheme. 3. MODELING 3.1. Simulation modeling Let us consider three model examples that illustrate the method’s performance on noisy data with some oscillations. 1) Oscillatory system with two periods yt = sin(πt/15) + sin(πt/20) + εt/2, where (εt) are independent identically distributed random variables with εt ∼ N (0, 1). A grid with t = 0, . . . , 299 of length 300 was taken, and a trajectory was simulated on it. To estimate the oscillation period, the method described in [2, 3] was applied. According to this method, the spectrum of the matrix log(A) is found. We assume that matrix A does not have multiple eigenvalues. Under this condition, oscillation with angular frequency ω will correspond to a pair of complex conjugate numbers z = α + iω and z̄ = α− iω, where the oscillation period is given by π/ω. In the first example, the estimated periods of oscillation were 30.49 and 39.93, while the original periods in the system before noise were 30 and 40 (see Fig. 3.1). In the two following examples, the estimated oscillation periods were the same as in the first one, up to rounding accuracy (see Fig. 3.2 and 3.3). 0 50 100 150 200 250 300 Time −3 −2 −1 0 1 2 3 W av e Filtration and forecast of a noisy wave Original Wave with Noise Predicted Wave Filtered Wave Separation line between in-sample and out-of-sample Fig. 3.1. Filtration and forecast of a noisy wave. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 12 A. BELYAKOV, A. KURBATSKII, A. SIDORENKO 0 50 100 150 200 250 300 Time −4 −2 0 2 4 6 W av e Filtration and foreca t of a noi y wave Original Wave with Noi e Predicted Wave Filtered Wave Separation line between in-sample and out-of-sample Fig. 3.2. Filtration and forecast of a noisy wave with exponents. 0 50 100 150 200 250 300 Time 0 5 10 15 20 W av e Filtration and forecast of a noisy wave Original Wave with Noise Predicted Wave Filtered Wave Separation line between in-sample and out-of-sample Fig. 3.3. Filtration and forecast of a noisy wave with exponential trend. 2) Oscillatory system with variable amplitude yt = e0.005t sin(πt/15) + e−0.001t sin(πt/20) + εt/2. 3) System with oscillations and exponential trend yt = e0.01t + e0.005t sin(πt/15) + e−0.001t sin(πt/20) + εt/2. 3.2. Electricity price modeling For modeling based on real data, we will consider electricity prices. The electricity market is of interest due to the presence of high volatility, multiple seasonality and calendar effects. In [11], it is shown that hourly electricity prices exhibit intraday, weekly, and monthly periodicities, as well as variable volatility. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) A STATE SPACE FILTERING-BASED APPROACH FOR PRICE PREDICTION 13 We use daily NordPool system data prices† from 2016 to 2020, as considered in the paper [9], because it serves as the unconstrained market clearing reference price for the European Nordic region. Daily prices were transformed into $/kWh. The sample consists of 1170 observations from January 01, 2016, to June 26, 2020. For out-of-sample forecasting, a rolling window of 22 ∗ 6 days (number of working days in a month multiplied by the number of months) is used (see Fig. 3.4). 20 16 -01 -04 20 16 -08 -01 20 17 -02 -27 20 17 -09 -25 20 18 -04 -23 20 18 -11 -19 20 19 -06 -17 20 20 -01 -13 Date 0.00 0.02 0.04 0.06 0.08 Pr ice Daily Electricity Prices Data Model in-sample prediction Model out-of-sample prediction Separation line between in-sample and out-of-sample Fig. 3.4. Electricity price with the model filtration and 1-day-ahead forecasts with a 6 month floating window. The estimated periods on the interval 2016-01-01 to 2016-06-30 are 11.22 and 22.34. To determine the state space dimension k, we used the mean absolute error for 1, 5, 22 days in a rolling-sample scheme with an interval size for modeling of 226 days (half a year). From the graph, it can be observed that the optimal dimensions for forecasting are k = 5 for daily and weekly predictions (see Fig. 3.6 and 3.7). It should be noted that for large dimensions k, out-of-sample errors increase, although in-sample errors decrease. A comprehensive review of literature on electricity price forecasting (EPF) methods and their comparison from 2012 to 2022 is presented in the article [14]. Notably, during relatively stable times, when conditions for building traditional econometric models are met, such methods work well and demonstrate high predictive characteristics. Even considering high volatility, the combined ARMA-type model and GARCH-type model are still applied [4, 6, 11]. To make a comparison with ARMA-type models, for which we selected a stationary series from January 01, 2016, to June 30, 2016. The results of corresponding tests are provided in table 3.1. Table 3.1. Stationarity tests results Test Statistic p-value ADF −5.3567 4.17× 10−6 KPSS 0.2103 > 0.1 DF-GLS −4.6647 5.38× 10−6 †The Nord Pool exchange has free data https://www.nordpoolgroup.com/en/Market-data1/Dayahead/Area- Prices/SYS1/Daily/?view=table Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 14 A. BELYAKOV, A. KURBATSKII, A. SIDORENKO Besides, an alternative and well-established data filtering method with seasonality, Holt– Winters exponential smoothing, was considered. Fig. 3.5 shows the mean absolute error dependencies for our method for different values of k, as well as for the ARMA(1,1) model, which proved to be the best model of this class on the training set, and the ETS model using Holt–Winters’ method. One-day-ahead, weekly, and monthly forecasts were made using a rolling-window scheme for all models. Over an extended period, the markets often exhibit a complex nonlinear structure as well as natural non-stationarity. An attempt to forecast the price for the next six months is shown in Fig. 3.5. It can be seen that our presented model captures the trend, while the ARMA model and the Holt–Winters’ method, even though it gives more accurate in-sample forecast, degenerate into a horizontal line. The fact that these models struggle to predict the behavior of such data has long been known, so researchers seek approaches to improve their quality. A survey on subspace methods for estimating linear dynamic models can be found in [1]. 20 15 -12 -31 20 16 -02 -19 20 16 -04 -09 20 16 -05 -29 20 16 -07 -18 20 16 -09 -06 20 16 -10 -26 20 16 -12 -15 Date 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 Pr ice Forecast and filtration of the State Space model Data State Space filtration ETS filtration ETS forecast ARIMA forecast State Space forecast Separation line between in-sample and out-of-sample Fig. 3.5. State-space filtration and forecast and comparison with ARIMA forecast and Holt–Winters ETS filtration. The periods calculated by the State Space model are 11.22 and 22.34 days. Fig. 3.6, 3.7 and 3.8 show how the MAE of considered method changes depending on the dimension of the state space k. For the short-term forecasts, the optimal value of k is 5, whereas for monthly forecasts, it is equal to 1. The implementation of our forecasting and filtering method in the Python programming language can be found at the following link [7]. 4. CONCLUSION Methods of forecasting based on the state space representation of a system have a wide range of applications for estimating various dynamic systems. However, in different research contexts and on different types of data, choosing the appropriate method is a non-trivial task, considering the inherent advantages and disadvantages of each method. Our proposed forecasting method and an improved filtering method allow working with ill-conditioned system matrices. The ability to identify periods of fluctuations is also of interest for understanding the behavior of cyclical indicators, particularly economic ones. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) A STATE SPACE FILTERING-BASED APPROACH FOR PRICE PREDICTION 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Hidden dimension 0.002 0.003 0.004 0.005 0.006 M AE Electricity MAE, 1-day(s)-ahead forecasts, rolling window with length 22 * 6 days State space ARIMA (1, 0, 1) ETS Fig. 3.6. 1-day-ahead MAE over 2016 (6 month window). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Hidden dimension 0.0025 0.0030 0.0035 0.0040 0.0045 0.0050 0.0055 0.0060 0.0065 M AE Electricity MAE, 5-day(s)-ahead forecasts, rolling window with length 22 * 6 days State space ARIMA (1, 0, 1) ETS Fig. 3.7. 5-days-ahead MAE over 2016 (6 month window). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Hidden dimension 0.004 0.006 0.008 0.010 0.012 0.014 0.016 M AE Electricity MAE, 22-day(s)-ahead forecasts, rolling window with length 22 * 6 days State space ARIMA (1, 0, 1) ETS Fig. 3.8. 22-days-ahead MAE over 2016 (6 month window). To illustrate the method, simulation modeling was conducted on typical types of oscillatory systems, and the forecast quality was evaluated with different horizons for electricity prices. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) 16 A. BELYAKOV, A. KURBATSKII, A. SIDORENKO During the comparison of model forecasts on electricity price data, it was found that the forecasts of the proposed method yield a comparable mean absolute error to the forecasts of traditional econometric models such as ARMA on stationary series and a well-known seasonal data filtering method like Holt–Winters’ method. The mean absolute error was calculated for forecasts with a rolling window of 6 months. It is worth noting that the method performs well in handling multiple seasonality and exponential trends inherent in data with such a complex structure as electricity prices. Additionally, oscillation periods were calculated, and they turned out to be 11 and 22 days, corresponding to two-week and monthly seasonality. ACKNOWLEDGEMENTS The research was conducted with financial support of the Russian Science Foundation grant No. 19-11-00223. REFERENCES 1. Bauer, D. (2005). Estimating linear dynamical systems using subspace methods. Econometric Theory, 21(1), 181-211. doi:10.1017/S0266466605050127 2. Belyakov, A. O., & Blazhennova-Mikulich, L. Yu. (2005) Identifikatsiya inertsionnoy matritsy konservativnoy kolebatelnoy sistemy [Identification of the inertia matrix of a conservative oscillating system]. Vestnik Moskovskogo universiteta. Seriya 1. Matematika. mekhanika, 3, 25–28. 3. Belyakov, A. O., & Seiranyan, A. P. (2008) Determining the moments of inertia of large bodies from vibrations in elastic suspension. Mechanics of Solids, 43, 205–217. 4. Bill´e, A.G., Gianfreda, A., Del Grosso, F., Ravazzolo, F. (2023). Forecasting electricity prices with expert, linear, and nonlinear models. International Journal of Forecasting, 39(2), 570-586. https://doi.org/10.1016/j.ijforecast.2022.01.003 5. Clark, P. K. (1987) The cyclical component of U. S. economic activity. The Quarterly Journal of Economics, 102(4). https://doi.org/10.2307/1884282 6. Girish, G.P., 2016. Spot electricity price forecasting in Indian electricity market using autoregressive-GARCH models. Energy Strategy Reviews 11–12, 52–57. https://doi.org/ 10.1016/j.esr.2016.06.005 7. Github. (2023, November). Subspace method. [Online]. Available https://github.com/ArturSidorenko/Subspace method 8. Fama, E. F., & Gibbons, M. R. (1982) Inflation, real returns and capital investment. Journal of Monetary Economics, 9(3). https://doi.org/10.1016/0304-3932(82)90021-6 9. Fantazzini D., Kolodin N. (2020) Does the hashrate affect the bitcoin price? Journal of Risk and Financial Management, 13(11), 1-–29. 10. Kung S. Y. (1978) A new identification and model reduction algorithm via singular value decomposition, Proc. 12th Asilomar Conf. on Circuits, Systems and Computer, 705-714. 11. Liu, Heping & Shi, Jing. (2013). Applying ARMA–GARCH approaches to forecasting short-term electricity prices. Energy Economics, 37, 152-–166. https://doi.org/10.1016/ j.eneco.2013.02.006 12. Meinhold, R. J., & Singpurwalla, N. D. (1983) Understanding the Kalman filter. The American Statistician, 37(2). https://doi.org/10.2307/2685871 13. Mergner, S. (2009) Applications of state space models in finance. An empirical analysis of the time-varying relationship between macroeconomics, fundamentals and Pan- European industry portfolios. Göttingen: Universitätsverlag Göttingen. 14. Shanglei Chai, Qiang Li, Mohammad Zoynul Abedin, Brian M. Lucey. (2024) Forecasting electricity prices from the state-of-the-art modeling technology and the Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) doi:10.1017/S0266466605050127 https://doi.org/ 10.1016/j.ijforecast.2022.01.003 https://doi.org/10.2307/1884282 https://doi.org/10.1016/j.esr.2016.06.005 https://doi.org/10.1016/j.esr.2016.06.005 https://doi.org/10.1016/0304-3932(82)90021-6 https://doi.org/10.1016/j.eneco.2013.02.006 https://doi.org/10.1016/j.eneco.2013.02.006 https://doi.org/10.2307/2685871 A STATE SPACE FILTERING-BASED APPROACH FOR PRICE PREDICTION 17 price determinant perspectives. Research in International Business and Finance, 67, Part A, 102132. https://doi.org/10.1016/j.ribaf.2023.102132 15. Triantafyllopoulos, K. (2021) Bayesian inference of state space models. Kalman filtering and beyond. Cham, Switzerland: Springer. https://doi.org/10.1007/ 978-3-030-76124-0 16. Viberg, M. (1995) Subspace-based methods for the identification of linear time- invariant systems. Automatica, 31(12), 1835–1851. http://dx.doi.org/10.1109/ACC. 2005.1470314 17. Zeng, Y., & Wu, S. (2013) State-space models. Applications in economics and finance. New York: Springer Science & Business Media. Copyright © 2023 ASSA. Adv Syst Sci Appl (2023) https://doi.org/10.1016/j.ribaf.2023.102132 https://doi.org/10.1007/978-3-030-76124-0 https://doi.org/10.1007/978-3-030-76124-0 http://dx.doi.org/10.1109/ACC.2005.1470314 http://dx.doi.org/10.1109/ACC.2005.1470314 Introduction Basics Modeling Simulation modeling Electricity price modeling Conclusion