Energy Management Strategy of Integrated Electricity-Heat Energy System Based on Federated Reinforcement Learning

doi:10.16183/j.cnki.jsjtu.2022.418

Abstract

Abstract:

The energy management of the electricity-heating integrated energy system (IES) is related to the economic benefits and multi-energy complementary capabilities of a park, but it faces the challenges of the randomness of renewable energy and the uncertainty of load. First, in this paper, a mathematical model of the energy management problem for the electricity-heating IES is conducted, and each energy supply subsystem is empowered as an agent. Based on the deep deterministic policy gradient (DDPG) algorithm, a system energy management model is established that comprehensively considers the real-time energy load of the subsystem, the time-of-use pricing, and the output of each equipment. Then, the federated learning technology is used to interact with the gradient parameters of the energy management model of the three subsystems during the training process to synergistically optimize the training effect of the model, which can protect the data privacy of each subsystem while breaking the data barriers. Finally, an example analysis verifies that the proposed federated-DDPG energy management model can effectively improve the economic benefits of the park-level IES.

Key words: integrated energy system (IES), federated learning, energy management, deep deterministic policy gradient (DDPG)

CLC Number:

TM732

WANG Jinfeng, WANG Qi, REN Zhengmou, SUN Xiaochen, SUN Yi, ZHAO Yiyi. Energy Management Strategy of Integrated Electricity-Heat Energy System Based on Federated Reinforcement Learning[J]. Journal of Shanghai Jiao Tong University, 2024, 58(6): 904-915.

Figures/Tables 12

Fig.1

Fig.2

Fig.3

Tab.1

Tab.2

Tab.3

Tab.4

Fig.4

Fig.5

Fig.6

Fig.7

Tab.5

References 29

[1]	苏慧玲, 杨世海, 陈铭明. 考虑能源效率的综合能源系统多目标优化调度[J]. 电力系统及其自动化学报, 2022, 34(2): 130-136.
	SU Huiling, YANG Shihai, CHEN Mingming. Multi-objective optimal scheduling of integrated energy system considering energy efficiency[J]. Proceedings of the CSU-EPSA, 2022, 34(2): 130-136.
[2]	董文杰, 田廓, 陈云斐, 等. 能源互联网下基于博弈与证据理论的综合能源系统评价方法研究[J]. 智慧电力, 2020, 48(7): 73-80.
	DONG Wenjie, TIAN Kuo, CHEN Yunfei, et al. Evaluation method of comprehensive energy system based on game theory & evidence theory under energy Internet[J]. Smart Power, 2020, 48(7): 73-80.
[3]	朱志芳, 许苑, 岑海凤, 等. 考虑需求侧响应的园区综合能源系统优化配置[J]. 智慧电力, 2022, 50(1): 37-44.
	ZHU Zhifang, XU Yuan, CEN Haifeng, et al. Optimal configuration of park-level integrated energy system considering demand response[J]. Smart Power, 2022, 50(1): 37-44.
[4]	李驰宇, 高红均, 刘友波, 等. 多园区微网优化共享运行策略[J]. 电力自动化设备, 2020, 40(3): 29-36.
	LI Chiyu, GAO Hongjun, LIU Youbo, et al. Optimal sharing operation strategy for multi park-level microgrid[J]. Electric Power Automation Equipment, 2020, 40(3): 29-36.
[5]	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.
	XIONG Luolin, MAO Shuai, TANG Yang, et al. Reinforcement learning based integrated energy system management: A survey[J]. Acta Automatica Sinica, 2021, 47(10): 2321-2340.
[6]	CARLI R, DOTOLI M. Decentralized control for residential energy management of a smart users microgrid with renewable energy exchange[J]. CAA Journal of Automatica Sinica, 2019, 6(3): 641-656.
[7]	FARROKHIFAR M, AGHDAM F H, ALAHYARI A, et al. Optimal energy management and sizing of renewable energy and battery systems in residential sectors via a stochastic MILP model[J]. Electric Power Systems Research, 2020, 187: 106483.
[8]	ALIPOUR M, ZARE K, ABAPOUR M. MINLP probabilistic scheduling model for demand response programs integrated energy hubs[J]. IEEE Transactions on Industrial Informatics, 2018, 14(1): 79-88.
[9]	MOSER A, MUSCHICK D, GÖLLES M, et al. A MILP-based modular energy management system for urban multi-energy systems: Performance and sensitivity analysis[J]. Applied Energy, 2020, 261: 114342.
[10]	杨家豪. 区域综合能源系统冷-热-电-气概率多能流计算[J]. 电网技术, 2019, 43(1): 74-82.
	YANG Jiahao. Calculation of cold-heat-electricity-gas probabilistic multi-energy flow in regional comprehensive energy system[J]. Power System Technology, 2019, 43(1): 74-82.
[11]	翟晶晶, 吴晓蓓, 傅质馨, 等. 考虑需求响应与光伏不确定性的综合能源系统鲁棒优化[J]. 中国电力, 2020, 53(8): 9-18.
	ZHAI Jingjing, WU Xiaobei, FU Zhixin, et al. Robust optimization of integrated energy systems considering demand response and photovoltaic uncertainty[J]. Electric Power, 2020, 53(8): 9-18.
[12]	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 46(7): 1301-1312.
	SUN Changyin, MU Chaoxu. Important scientific problems of multi-agent deep reinforcement learning[J]. Acta Automatica Sinica, 2020, 46(7): 1301-1312.
[13]	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803.
	LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Research on energy management and optimization strategy of micro-energy network based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794-3803.
[14]	乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-832.
	QIAO Ji, WANG Xinying, ZHANG Qing, et al. Optimal dispatching of electric-gas integrated energy system based on flexible actuator-judge deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-832.
[15]	ZHANG B, HU W H, LI J H, et al. Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: Deep reinforcement learning approach[J]. Energy Conversion & Management, 2020, 220: 113063
[16]	XI L, YU L, XU Y C, et al. A novel multi-agent DDQN-AD method-based distributed strategy for automatic generation control of integrated energy systems[J]. IEEE Transactions on Sustainable Energy, 2020, 11(4): 2417-2426.
[17]	陈明昊, 孙毅, 胡亚杰, 等. 基于纵向联邦强化学习的居民社区综合能源系统协同训练与优化管理方法[J]. 中国电机工程学报, 2022, 42(15): 5535-5549.
	CHEN Minghao, SUN Yi, HU Yajie, et al. The collaborative training and management-optimized method for residential integrated energy system based on vertical federated reinforcement learning[J]. Proceedings of the CSEE, 2022, 42(15): 5535-5549.
[18]	SAPUTRA Y M, HOANG D T, NGUYEN D N, et al. Energy demand prediction with federated learning for electric vehicle networks[C]//2019 IEEE Global Communications Conference. Waikoloa, USA: IEEE, 2019: 1-6.
[19]	LEE S, CHOI D H. Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources[J]. IEEE Transactions on Industrial Informatics, 2022, 18(1): 488-497.
[20]	周长城, 马溪原, 郭晓斌, 等. 基于主从博弈的工业园区综合能源系统互动优化运行方法[J]. 电力系统自动化, 2019, 43(7): 74-80.
	ZHOU Changcheng, MA Xiyuan, GUO Xiaobin, et al. Leader-follower game based optimized operation method for interaction of integrated energy system in industrial park[J]. Automation of Electric Power Systems, 2019, 43(7): 74-80.
[21]	吕佳炜, 张沈习, 程浩忠. 计及热惯性和运行策略的综合能源系统可靠性评估方法[J]. 电力系统自动化, 2018, 42(20): 9-16.
	LYU Jiawei, ZHANG Shenxi, CHENG Haozhong. Reliability evaluation of integrated energy system considering thermal inertia and operation strategy[J]. Automation of Electric Power Systems, 2018, 42(20): 9-16.
[22]	白庆林, 张培山, 张林江, 等. 城市燃气SCADA及信息管理系统设计[J]. 自动化与仪表, 2009, 24(4): 47-50.
	BAI Qinglin, ZHANG Peishan, ZHANG Linjiang, et al. Design of information management system for urban natural gas SCADA system[J]. Automation & Instrumentation, 2009, 24(4): 47-50.
[23]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[24]	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California, USA: ACM, 2017: 6382-6393.
[25]	KOBAYASHI T, ILBOUDO W E L. T-soft update of target network for deep reinforcement learning[J]. Neural Networks, 2021, 136: 63-71. doi: 10.1016/j.neunet.2020.12.023 pmid: 33450653
[26]	MCKENNA E, THOMSON M. High-resolution stochastic integrated thermal-electrical domestic demand model[J]. Applied Energy, 2016, 165: 445-461.
[27]	THOMAS D, D’HOOP G, DEBLECKER O, et al. An integrated tool for optimal energy scheduling and power quality improvement of a microgrid under multiple demand response schemes[J]. Applied Energy, 2020, 260: 114314.
[28]	AYYADI S, BILIL H, MAAROUFI M. Optimal charging of Electric Vehicles in residential area[J]. Sustainable Energy, Grids & Networks, 2019, 19: 100240.
[29]	SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. 2nd ed. Cambridge, Massachusetts: The MIT Press, 2018.

供能子系统	所包含设备
MG₁	风力发电机、光伏、CHP机组、电、热储能系统、燃气锅炉、电热锅炉及电、热负荷
MG₂	风力发电机、光伏、CHP机组、电、热储能系统、燃气锅炉及电、热负荷
MG₃	风力发电机、光伏、电、热储能系统、燃气锅炉、电热锅炉及电、热负荷

设备	P^min, H^min/MW	P^max, H^max/MW
CHP机组	0	1.2
电热锅炉	0	1.4
燃气锅炉	0	1.2
蓄电池	-0.2	0.2

参数	数值	参数	数值
η_CHP	0.32	η_EB	0.94
β	1.25	η_GB	0.82
η_HE	0.73	C_Ncurt	0.20

时段	购电电价/ [元·(kW·h)^-1]	售电电价/ [元·(kW·h)^-1]
峰时段	0.922	0.500
平时段	0.540	0.200
谷时段	0.382	0

优化方法	成本/元			有效训练时长/s
优化方法	MG₁	MG₂	MG₃	有效训练时长/s
传统方法	19342.00	16786.00	49560.00	14228.00
MADDPG	18424.00	16629.00	47892.00	10634.20
F-DDPG	12973.02	13845.09	41469.00	6121.80