CHEMICAL ENGINEERING TRANSACTIONS VOL. 81, 2020 A publication of The Italian Association of Chemical Engineering Online at www.cetjournal.it Guest Editors: Petar S. Varbanov, Qiuwang Wang, Min Zeng, Panos Seferlis, Ting Ma, Jiří J. Klemeš Copyright © 2020, AIDIC Servizi S.r.l. ISBN 978-88-95608-79-2; ISSN 2283-9216 Research on Photovoltaic Energy Storage System and Supply-Side Power Dispatch Model in Paper Mill Ying Gao, Jigeng Li, Mengna Hong, Yi Man* State Key Laboratory of Pulp and Paper Engineering, South China University of Technology, Guangzhou, 510640, China manyi@scut.edu.cn With the increased awareness of environmental protection and the establishment of the spot market of electric power, renewable energy plays a more and more important role in energy structure adjustment. But the intermittence and instability of renewable energy often result in energy waste. Micro-grid energy storage can effectively improve the utilization rate, economy and reliability of renewable energy. To ensure the economical operation of the micro-grid system and improve the load curve, a two-stage optimal dispatch model was established based on non-dominated sorting genetic algorithm (NSGA-II) and deep Q-value reinforcement learning algorithm (DQN). In the first stage, to get the appropriate battery capacity, the minimum comprehensive operation cost and the minimum capacity fading of battery in the cycle life were selected as the optimization objectives. The optimal solution was calculated by using the improved non-dominated sorting genetic algorithm (NSGA-II). In the second stage, the DQN Algorithm was used to propose a scheduling strategy. Compared with the GA method, the DQN method was able to reduce the operating cost from 0.5829 % to 1.2294 %. The results showed that the method was feasible and effective. 1. Introduction Nowadays, the battery energy storage systems (BESS) with intermittent renewable energy systems is a promising technical solution to guarantee the security and reliability of the electric power system. Solar power has rapidly developed as a clean and pollution-free renewable energy and there are many discussions about harnessing solar power (Xu et al., 2020). According to the International Energy Agency (IEA), new installed capacity for renewable energy is expected to grow by nearly 12 % that is almost 200 GW this year. Global Photovoltaic generation is expected to grow by more than 17 % (Huang et al., 2020). However, the output of renewable energy power and distributed generation (DG) is variable or even random, making it difficult to be integrated into the power grid. To improve the market-based electricity quantity balance mechanism and price formation mechanism as well as promote the formation of a clean, low-carbon, safe and efficient energy system, a medium-and long-term electricity market supplemented by spot transactions will be established, following the laws of the market and the operation of the power system (Peng et al., 2018). The paper mill is an energy- intensive manufacturing industry (Zeng et al., 2018). It is vital for paper mills to estimate the amount of electricity need to buy from power grid the next state by predicting demand load as well as finding the way to cut the peak and fill the valley. According to this forecasting data, paper mills can sign agreements with grid-side companies. In recent years, BESS plays a vital role in storing superfluous solar energy to save on operating costs. Scholars from different countries have conducted a variety of studies applying BESS to the electric system. There are several types of batteries available for the power sector, including pumping energy storage, electrochemical energy storage, supercapacitor, superconducting magnetic energy storage, flywheel versus battery, compressed air, hydrogen storage, and methane storage (Esparcia et al., 2019). In the above types of batteries, electrochemical energy storage technology is an effective means to solve typical volatility and randomness of renewable energy (Yao, 2018). One of the fundamental issues of power system operation is economic emission load dispatch problem (Ma et al., 2017). As the battery is essential in a microgrid, the capacity reduction of the battery is also an objective function considered in the power dispatching model. The degradation of the Li-ion cell was taken into DOI: 10.3303/CET2081208 Paper Received: 05/06/2020; Revised: 06/07/2020; Accepted: 07/07/2020 Please cite this article as: Gao Y., Li J., Hong M., Man Y., 2020, Research on Photovoltaic Energy Storage System and Supply-Side Power Dispatch Model in Paper Mill, Chemical Engineering Transactions, 81, 1243-1248 DOI:10.3303/CET2081208 1243 consideration that the resistive film growth at the cell solid-electrolyte interphase and the consumption of active materials in the electrodes. Researchers have made much progress in economic emission load dispatch problem. A particle swarm optimization algorithm (Li et al., 2020), a multi-objective model with two objectives (Liu et al., 2020) and differential evolution algorithm (Qiao et al., 2020) were proposed in succession to solve the economic emission load dispatch problem. However, the electricity consumption in the papermaking process is not periodic and stable, but short-term electricity consumption is predictable (Hu et al., 2019). The parameters should be adaptive to accommodate the different period of the paper mill but the parameters used in the methods above were set by the author. DQN used Q-learning with deep neural networks for function approximation. The parameters of neural networks were updated every few steps (Nian et al., 2020). On the above premise, this paper proposed a two-stage power dispatching strategy to solve the economic problem of the power supply of the paper mill, and proved that the method is feasible and effective by simulation. 2. System structure and working principle The structure of the BESS system was composed of these components: LiFePO4 battery, photovoltaic system, converter, inverter and controller. The work mode of the system can be divided as follows: Mode 1: The battery energy storage system and the supply side could obtain electricity from the grid and the solar power plant. Mode 2: The solar power firstly satisfied the battery energy storage system and then the supply side. Mode 3: There was a controller to determine whether the solar power produced at that time charged the battery or provided for the power side, and the grid could not charge the battery energy storage system. Mode 4: Battery storage systems could obtain power only from the solar power plant, but the supply side could get from both the solar power plant and the grid. To support the modes of operation, the BESS installed could reasonably plan its charging and discharging action according to the remaining capacity during the process. During operation, the output of the solar power plant was limited by the maximum generation of the solar power plant and the local climate of the day. The output of solar power, the demand for the supply side and the mode joint the solution of charge and discharge at the time, might lead to an increase in operating cost. To minimize these operation costs, we should establish a proper cost model and adequate capacity planning of the BESS. 3. Operation cost model To minimize the operating cost of paper mills, this paper optimized the cost of BESS and the cost of buying power from grid or solar power plant to meet the need for supply-side. By comparing the operating costs based on different sizing of BESS, the proper battery capacity was obtained. This section first introduced the model of problem formation in section 3.1, then described the four battery pack installation methods of charging and discharging in section 3.2. 3.1 Construction of system operating cost model A mathematical model of the cost of electricity generation and emissions was transformed into a multi-objective optimization problem. To make the most of solar power and minimize the cost of system cost, the model was defined as having the following two objectives: 3.1.1 Cost model for electricity F1 = ∑ z1(t) + z2(t) - z3(t) t t=1 (1) where, z1 is the cost of buying power from the grid, z2 is the cost of power from the solar power plant, z3 is the cost of power from the battery storage system, and t is the running time of the system. 3.1.2 Cost model for BESS F2 = ∑ u1(t) + u2(t) + u3(t) t t=1 (2) where, u1 is discarding generated solar power, u2 is the operating cost of BESS, and u3 is the original investment of BESS. 1244 When the output of the solar power plant is greater than the sum of BESS and supply side can accept, the system must discard the excess solar power at a cost, the cost model for discarded generated solar power energy as follows: u1(t) = ∑ ∑ CdspEdsp(t) 24 t=1 D d=1 (3) where, Cdsp is the discarding solar power cost and Edsp is the electric quantity of discarding solar power per hour. The cost of BESS also included the cost of battery degradation and the average daily cost of the original purchase of the batteries. At present, research on mechanism model and semi-empirical can explain the battery degradation, but the mechanism model is more complex and has more parameters. The semi-empirical model is easier to be applied in engineering practice (Jin et al., 2019). This model used a semi-empirical model to estimate battery capacity degradation (Wang et al., 2011) as follows and parameters (Song et al., 2014) with the values shown in Table 1. Qloss = A0e -( Ea+B∙CRate RTbat ) (Ah) z (4) where, Ea is the activation energy (J/mol), R is the ideal gas constant, CRate is the absolute value of the battery charge and discharge current rate, Ah is the Ah-throughput, A0 is the pre-exponential coefficient, z is a second undetermined coefficient, B is a third undermined coefficient, and Tbat is the temperature of the battery in Kelvin (Liu et al.,2020). u2(t) = Cbatpricebat(2.738×10 -3 |Ibat |e - 15162-1516×𝐶𝑅𝑎𝑡𝑒(t) 2041.52 Qloss(t-1) -0.2136 (5) where, Cbat is the total capacity of the BESS (Wh), pricebat is the price of the battery (RMB/Wh), Ibat is the operating current. This paper is mainly focused on the performance of BESS, the accuracy is not very high, so the current chose the ideal current. The cycle times of LiFePO4 battery was estimated to be over 3,000, and it could be used from 10 to 15 y. In this study, the operating cycle of the BESS was set to 10 y. The origin cost of BESS was described as follows: u3(t) = price bat ∙Cbat∙t 5×365×24 (6) where, t is the operating time (h). 3.1.3 Dispatch model Getting the BESS capacity, DQN method was used to solve the dispatch problem. The import steps of DQN can be described as shown in Figure 1 (Mnih et al., 2013): Figure 1: Flow Chart of Deep Q-learning Algorithm Q is a function to estimate the action is useful or harmful to the environment. “s” is state and “a” is action. “s’” is the next state and “a’” is the following action. “θ” is Mean square error and loss function. “r” is a current return. In the DQN method, a network is used to calculate Q value. Though a similar network, the existing value network used the new parameters, the target value network used the parameters sometime before. The loss function of DQN is shown as follow: L(w) = E[(r +γ∙ max Q(s',a';θ)- Q(s, a; θ) 2 )] (7) 1245 where, γ is discount coefficient 3.2 Optimization objectives and constraints 3.2.1 Charge and discharge rate constraint The factor affects the rate of charge and discharge of the battery, first of all is the properties of the battery itself. The batteries worked in proper temperature, and its stable charge rate was set to 1 time its capacity (1 C charge rate), rate of discharge was set to 0.5 times its capacity (0.5 C discharge rate). At the same time, the power and voltage of supply-side influenced the action of the battery. Actual charge and discharge rate might be lower than the rated one. 3.2.2 BESS installed position constraint In Mode 1, if SOC was more than 90 %, the excess power provided for the supply side. If SOC was less than 10 %, the BESS was forced not to discharge. If the output of the solar power plant could not meet the demand of charging, the batteries charge from the grid. Considering the maximum and minimum of SOC, the initial battery capacity was supposed to be 10 % SOC. In Mode 2, if SOC was more than 90 %, the excess power would be discarded. If SOC was less than 10 %, the BESS was forced to charge. If the output could not meet the demand, the BESS remained the same. In Mode 3, if SOC was more than 90 %, the excess power was first satisfied the supply side, and then would be discarded. If SOC was less than 10 %, the BESS was forced not to discharge. If the output of the solar power plant could not meet the demand, the BESS remained the same. In Mode 4, if SOC was more than 90 %, the excess power would be discarded. If SOC was less than 10 %, the BESS was forced not to discharge. If the output of the solar power plant could not meet the demand, then the grid would fill in the gap. 3.2.3 Parameters In this paper, the parameters of the battery cell were as shown in Table 1. The charge and discharge actions were regarded as an individual in NSGA-II and coded in the form of a real matrix. For the NSGA-II method, the parameters selected in this paper are shown in Table 2. Table 1: Parameters of LiFePO4 Battery Cell Parameter Value Nominal voltage (V) 3.2 Capacity (Ah) 280 Operating temperature (℃) -20-45 Table 2: Parameters of NSGA-II Method Parameter Value Population size 200 Crossover fraction 0.4 Migration fraction 0.4 Generations 50 The parameters in Table 1 were used to calculate the capacity of BESS when the current in batteries was the same as the one in the supply side. The batteries had estimated cycle times when the operating temperature was within an appropriate range. As for the parameters in Table 2, they are used in the first stage and the optimal solution could be obtained better and faster by choosing appropriate parameters for NSGA-II Method. 4. Simulation results and analysis A paper mill needing 45 MWh/y and a solar power plant with a capacity of 7.3 MWh/y were selected for simulation. From the historical operational data of paper mill, the daily solar power output data and demand electric capacity of supply side in typical operating conditions were selected. It could be observed that solar power plants would discard redundant solar power when solar power was sufficient and only charge for BESS, while the case charging for the supplier would not occur. It could also be observed that paper mills need a lot of electric capacity, so the solar output was rarely higher than paper mills need. The current in the circuit was affected by both the charge and discharge rate of the battery as well as the flow on the supply side, finally, take the smallest of them. It was found that the current fluctuates in a short-range for 1246 the paper mill. In this paper, the average current over the past year was selected as the current of the supply- side in the calculation. To make the most of the output of the solar power plant, the day of maximum solar radiation was selected to be the subject in the first stage of the optimal dispatch model. Further, the grid price was divided into spiking, peak, flat section and low cost. From 19:00 to 21:00 the price is 1.0824 RMB/kWh for spiking time, from 8:00 - 10:00 and 15:00 - 18:00 it was 0.9004 RMB/kWh for peak time, from 23:00 to 6:00 the next day it was 0.4164 RMB/kWh for low price time, at the other time of the day it was 0.6644 RMB/kWh for flat section time. The efficiency of the converter was set to 95 %, and the effectiveness of the inverter was set to 99 %. Considering the influence of the depth of charge and discharge on battery life, the battery operating range was set to 10 % - 90 % SOC and the initial stage was set to 10 %. The costs of four modes were calculated then compared and simulated, the capacity of the BESS was set to 583 Wh, the minimum cost was in Mode 1. Figure 2 shows the costs of four modes in the day of maximum solar radiation. The cost of Mode 2 was the highest, and the one of Mode 1 was the lowest. Figure 2: Cost of Four Modes in The Day of Maximum Solar Radiation Figure 3: Reference Load, Actual Load and DQN load at Capacity = 3,413.76 kWh in Mode 1 Table 3: Comparison of operating costs for the DQN method and GA Parameter DQN GA Daily operating costs (RMB) 82,285 82,813 Cost reduction (%) -0.5829 -1.2294 When the capacity of BESS and the operating mode were calculated, DQN method was used to give a dispatch solution, as shown in Figure 3 and Table 3. It could be observed that the DQN method can reduce cost. Compared with the GA method, the DQN method was able to reduce the operating cost from 0.5829 % to 1.2294 %. Due to the limitation of the DQN method, it is difficult to obtain the optimal global solution. However, the DQN method was still better than the traditional heuristic method to some extent. 5. Conclusions In this paper, a component sizing method for supply side in paper mill based on dynamic programming was proposed. By calculating both the penalty power under different BESS capacity and the operating costs, it can be able to obtain an optimal BESS capacity. The simulation results show the following key findings. (1) By comparing the effects that different installed condition of battery has on the costs of BESS, it showed that the location of the installation affects profitability and obtain the optimal energy storage capacity. According to 1247 figure 2, when it was in Mode 2, the cost was far higher than in other modes, and the cost was lowest when it was in Mode 1. (2) It also showed that the energy storage capacity we obtained from (1) could reduce the cost of buying power. (3) Compared with the GA method, the DQN method was able to improve the operating cost reduction from 0.5829 % to 1.2294 %. By comparing the effects that different algorithms have on operating cost, it showed that though DQN method has a limitation, if the price of power energy is lower, BESS can effectively reduce the cost of using power. The proposed method was shown to adequately consider variations of the output of a solar power plant, need of supply-side in the paper mill and battery degradation caused by the irregular charging and discharging of the BESS. A reasonable BESS capacity configuration scheme can be achieved by our proposed method. In future work, to obtain a consistent capacity allocation scheme, different installed methods of BESS should be taken into account. References Esparcia E.A., Castro M.T., Buendia R.E., Ocon J.D., 2019, Long-discharge flywheel versus battery energy storage for microgrids: a techno-economic comparison, Chemical Engineering Transactions, 76, 949-954. Huang B., Zhang Y., Wang C., 2020, New energy development and issues in China during the 14th Five-Year Plan, Electric Power, 53, 1-9. Hu Y., Li J., Hong M., Ren J., Lin R., Liu Y., Liu M., Man Y., 2019, Short term electric load forecasting model and its verification for process industrial enterprises based on hybrid GA-PSO-BPNN algorithm—A case study of papermaking process, Energy, 170, 1215-1227. Jin X., Liu C., 2019, Physics-based control-oriented reduced-order degradation model for LiNiMnCoO2 - graphite cell, Electrochim, Acta 312, 188- 201. Li Y., Vilathgamuwa M., Choi S.S., Xiong B., Tang J., Su Y., Wang Y., 2020, Design of minimum cost degradation-conscious lithium-ion battery energy storage system to achieve renewable power dispatchability. Applied Energy, 260, 114282. Liu Y., Wu X., Du J., Song Z., Wu G., 2020, Optimal sizing of a wind-energy storage system considering battery life, Renewable Energy, 147, 2470-2483. Ma H., Yang Z., You P., Fei M., 2017, Multi-objective biogeography-based optimization for dynamic economic emission load dispatch considering plug-in electric vehicles charging, Energy, 135, 101 – 111. Mnih V., Kavukcuoglu K., Silver D., Graves A., Antonoglou I., Wierstra D., Riedmiller M., 2013, Playing atari with deep reinforcement learning, NIPS Deep Learning Workshop 2013, Lake Tahoe, Spain, , accessed 02/07/2020. Nian R., Liu J., Huang B., 2020, A review on reinforcement learning: Introduction and applications in industrial process control, Computers & Chemical Engineering, 1394, 106886. Peng X., Tao X., 2018, Cooperative game of electricity retailers in China's spot electricity market, Energy, 145, 15, 152-170. Qiao B., Liu J., 2020, Muti-objective dynamic economic emission dispatch based on electric and wind power integrated system using differential evolution algorithm, Renewable Energy, 154, 316-336. Song Z., Li J., Han X., Xu L., Lu L., Ouyang M., Hofman H., 2014, Multi-objective optimization of a semi-active battery/supercapacitor energy storage system for electric vehicles, Applied Energy, 135, 212- 224. Wang J., Liu P., Hicks-Garner J., Sherman E., Soukiazian S., Verbrugge M., Musser J., Finamore P., 2011, Cycle-life model for graphite-LiFePO4 cells, J. Power Sources, 196, 3942-3948. Xu M., Xie P., Xie B., 2020, Study of China's optimal solar photovoltaic power development path to 2050, Resources Policy, 65, 101541. Yao J., 2018, Analysis of influence of electrochemical battery energy storage regulation technology on stable operation of power system, Chemical Engineering Transactions, 67, 733-738. Zeng Z., Hong M., Li J., Man Y., Liu H., Lib Z., Zhang H., 2018, Integrating process optimization with energy- efficiency scheduling to save energy for paper mills, Applied Energy, 2251, 542-558. 1248