Academic Journal of Science and Technology ISSN: 2771-3032 | Vol. 10, No. 3, 2024 46 Hierarchical Energy Management Strategy Based On M‐ DQN for Fuel Cell Hybrid Electric Vehicle Pengfei Xu, Zhumu Fu College of Information Engineering, Henan University of Science and Technology, Luoyang, Henan, China Abstract: For fuel cell hybrid electric vehicles equipped with a fuel cell (FC), battery (BAT), and supercapacitor (UC), this paper proposes a hierarchical energy management strategy (EMS) based on the improved M-DQN algorithm to reduce hydrogen consumption, enhance the efficiency of the FC, and maintain the state of charge (SoC) of the BAT. Firstly, an adaptive fuzzy low-pass filter is employed to achieve the frequency decoupling of power demand, enabling the UC to provide/absorb the peak power demand. Secondly, an energy management framework based on M-DQN is designed, and the concept of an equivalent consumption minimization strategy is applied to construct the reward function, which includes penalty factors related to the efficiency of the FC and SoC deviation of the BAT, aiming at optimizing the power allocation between the FC and BAT. Simulations and platform tests were conducted under various typical driving cycles. The results indicate that, compared with the traditional M-DQN-based EMS, the proposed EMS can significantly maintain the SoC of the BAT, improve the efficiency of the FC, reduce hydrogen consumption, and even achieve an average improvement of 5.2% in fuel economy. Keywords: Deep reinforcement learning , FCHEV , Energy management strategy. 1. Introduction Relative to conventional fuel cell vehicles, fuel cell hybrid electric vehicles (FCHEVs) that utilize a combination of batteries (BAT) and ultracapacitors (UC) have shown enhanced capabilities in terms of dynamic performance and fuel cell (FC) efficiency, while also compensating for the deficiencies in power and fuel economy inherent to vehicles powered by a single energy source[1-2]. Selecting an optimal energy management strategy (EMS) is critical to effectively manage the distribution of power among the different energy systems, thereby improving the vehicle's fuel efficiency and extending the operational life of the energy sources[3-4]. Recent advancements in reinforcement learning (RL) are being applied to EMS for vehicles to improve fuel efficiency. Despite its effectiveness, the conventional Q-learning approach faces challenges with high-dimensional, continuous state-action spaces, known as the “curse of dimensionality” [5- 9]. In this paper, we introduce the Munchausen-DQN (M- DQN) for energy management in FCHEV, aimed at reducing value function overestimation. A fuzzy-controlled low-pass filter segregates power demand into frequency bands, with ultracapacitors addressing the high-frequency needs. The M- DQN reward function, inspired by the equivalent consumption minimum strategy (ECMS), facilitates effective power distribution between the battery and fuel cell. Additionally, dynamic programming enhances M-DQN's learning during offline training. This approach is validated through simulation and practical experimentation. 2. Energy Management System Figure 1 in this paper outlines the FCHEV's topology, where the FC serves as the primary power source, connected to the DC bus via a unidirectional DC/DC converter for stable power output. The BAT and UC, as secondary storage units, are linked through bidirectional converters for FC protection, braking energy recuperation, and transient power compensation, enhancing the vehicle's power efficiency and economy. Figure 1. The topology of three-energy-source hybrid electric vehicle 47 3. Hierarchical Energy Management Strategy Based on M-DQN Taking into account the distinct characteristics of each energy source, a hierarchical energy management approach utilizing Munchausen-DQN is developed. An adaptive fuzzy filter first directs high-frequency power demands to the UC. Subsequently, a framework for energy management rooted in Munchausen-DQN is formulated. Dynamic programming aids in offline network training, enabling the acquisition of an ongoing optimal strategy for power distribution between the FC and BAT. 3.1. Fuzzy-based adaptive filter The peak power demands within driving cycles can lead to irreversible harm to the FC and BAT. To safeguard these components from peak power impacts, an adaptive low-pass fuzzy filter is employed to isolate the high-frequency power needs, which the UC is capable of managing. This approach involves using the adaptive low-pass fuzzy filter to effectively mitigate high-frequency power demands. he adaptive low- pass fuzzy filter can be expressed as: 1 G( ) 1 1 f s s    (1) where f means the regulating frequency, calculated by a fuzzy inference system (FIS) according to demandP , ucSoC and BATSoC . 3.2. Energy management strategy based on the M-DQN The primary objective of this paper is to identify an optimal energy management strategy that minimizes fuel consumption, enhances the efficiency of FC, and preserves the SoC of the BAT, framing vehicle energy management as a multi-objective optimization challenge. To circumvent the extensive computational demands typically associated with solving such problems, we employ the concept of the ECMS for the formulation of the reward function, which can be expressed as follows: . . . min ( ) ( ) ( ) ( ) ( ) ( ) min 0 ( ) total FC FC BAT BAT UC UC BAT BAT ref BAT BAT ref BAT BAT BAT ref H t H t H t H t SoC t SoC SoC t SoC SoC SoC t SoC             (2) where ( )totalH t means the total equivalent hydrogen consumption, ( )FCH t is the hydrogen consumption of the FC, and ( )BATH t and ( )UCH t mean the equivalent hydrogen consumption of BAT and UC, respectively, ( )FCH t and ( )UCH t are presented in [21]. Since the goal of M-DQN-based DRL is to maximize the cumulative reward, the negative value with ( )totalH t and SoC is taken as the reward, which is expressed as follows: 2[ ( ) ( ) ]totalR H t SoC    (3) where  is the penalty factor to ensure that the value of Δ BATSoC is as small as possible. M-DQN is a reinforcement learning algorithm that improves upon the traditional Soft deep Q-Network (S-DQN) by introducing the concept of self-motivation. It incorporates an additional term, the Munchausen term, into the calculation of the Q-value, which is based on the logarithm of the probability of the current action. This allows the algorithm to consider its own choices during the learning process. This approach effectively reduces the issue of overestimation and enhances the algorithm's stability and performance in complex environments. The target Q-value and loss function are expressed as follows:         t t t 1 t 1 t 1 a A R ln a s a s Q s ,a ln a s                    (4) 4. Simulation and Analysis In this section, to demonstrate the advantages of M-DQN in offline learning and training, we selected the conventional S-DQN as a competitor. We compared the changes in average losses during the offline network training, as shown in Figure 2. The figure reveals that, compared to the traditional S-DQN algorithm, the M-DQN algorithm has a faster convergence rate in its loss curve and exhibits better convergence performance. Figure 2. The loss curve of the M-DQN algorithm 48 Figure 3. The simulation results of UDDS. Figure 3 presents the simulation results of the proposed strategy under a test driving cycle, where Figure 3 depicts the speed variation graph lasting approximately 1400 seconds during the test driving cycle. It also shows the power distribution of each energy source and the changes in the SoC of the lithium battery and supercapacitor under the proposed strategy. From the figure, it can be observed that the power provided by the fuel cell is relatively low and stable, which better protects its lifespan; the supercapacitor handles/absorbs most of the peak power, resulting in significant changes in its SoC, with the supercapacitor absorbing a considerable amount of peak power, causing a sudden increase in the system's chip in a short period. The lithium battery mainly compensates for the remaining power demand, with power fluctuations within an acceptable range. Compared to the traditional S-DQN strategy, under this strategy, the SoC of the lithium battery declines faster, and the average efficiency of the fuel cell is higher, fully demonstrating that the M-DQN strategy can maintain the SoC of the lithium battery, enhance the efficiency of the fuel cell, and improve the fuel economy of the fuel cell. 5. Conclusion This paper designs a power-layered continuous control EMS system for FCHEV, addressing the distinct physical characteristics of its three energy sources, based on the M- DQN algorithm. By embracing the concept of the ECMS, a multi-objective optimization function is formulated. In comparison to the traditional S-DQN energy management strategy, the M-DQN algorithm more effectively maintains the SoC of the lithium battery, enhances the efficiency of the fuel cell, significantly reduces the consumption of equivalent hydrogen fuel, and improves fuel economy by 5.2%. References [1] Ferrara A, Jakubek S, Hametner C. Energy management of heavy-duty fuel cell vehicles in real-world driving scenarios: Robust design of strategies to maximize the hydrogen economy and system lifetime[J]. Energy Conversion and Management, 2021, 232: 113795. [2] Kasimalla V K R, Velisala V. A review on energy allocation of fuel cell/battery/ultracapacitor for hybrid electric vehicles[J]. International Journal of Energy Research, 2018, 42(14): 4263- 4283. [3] Y. Hames, K. Kaya, E. Baltacioglu, A. Turksoy, Analysis of the control strategies for fuel saving in the hydrogen fuel cell vehicles[J]. International Journal of Hydrogen Energy, 2018, 43: 10810–10821. [4] Tran D D, Vafaeipour M, El Baghdadi M, et al. Thorough state- of-the-art analysis of electric and hybrid vehicle powertrains: Topologies and integrated energy management strategies[J]. Renewable and Sustainable Energy Reviews, 2020, 119: 109596. [5] Zou Y, Liu T, Liu D, et al. Reinforcement learning-based real- time energy management for a hybrid tracked vehicle[J]. Applied energy, 2016, 171: 372-382. [6] Sun H, Fu Z, Tao F, et al. Data-driven reinforcement-learning- based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles[J]. Journal of Power Sources, 2020, 455: 227964. [7] Ganesh A H, Xu B. A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution[J]. Renewable and Sustainable Energy Reviews, 2022, 154. [8] Liu T, Zou Y, Liu D, et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle[J]. IEEE Transactions on Industrial Electronics, 2015, 62(12): 7837-7846. [9] Ganesh A H, Xu B. A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution[J]. Renewable and Sustainable Energy Reviews, 2022, 154: 111833.