基于联邦强化学习的电热综合能源系统能量管理策略

doi:10.16183/j.cnki.jsjtu.2022.418

摘要/Abstract

摘要：

电热综合能源系统(IES)的能量管理关系到园区的经济效益与多能互补能力,但面临新能源出力随机性和用户负荷不确定性的挑战.首先,构建电热IES能量管理问题的数学模型,将各供能子系统赋能为智能体,基于深度确定性策略梯度(DDPG)算法建立综合考虑子系统实时用能负荷、分时电价及各设备出力的系统能量管理模型.然后,采用联邦学习技术,在训练过程中交互3个子系统的能量管理模型梯度参数对模型的训练效果进行协同优化,打破数据壁垒的同时保护各子系统数据隐私.最后,通过算例分析验证了所构建基于联邦学习框架的DDPG能量管理模型能有效提升园区IES经济效益.

关键词: 综合能源系统, 联邦学习, 能量管理, 深度确定性策略梯度

Abstract:

The energy management of the electricity-heating integrated energy system (IES) is related to the economic benefits and multi-energy complementary capabilities of a park, but it faces the challenges of the randomness of renewable energy and the uncertainty of load. First, in this paper, a mathematical model of the energy management problem for the electricity-heating IES is conducted, and each energy supply subsystem is empowered as an agent. Based on the deep deterministic policy gradient (DDPG) algorithm, a system energy management model is established that comprehensively considers the real-time energy load of the subsystem, the time-of-use pricing, and the output of each equipment. Then, the federated learning technology is used to interact with the gradient parameters of the energy management model of the three subsystems during the training process to synergistically optimize the training effect of the model, which can protect the data privacy of each subsystem while breaking the data barriers. Finally, an example analysis verifies that the proposed federated-DDPG energy management model can effectively improve the economic benefits of the park-level IES.

Key words: integrated energy system (IES), federated learning, energy management, deep deterministic policy gradient (DDPG)

中图分类号:

TM732

王金锋, 王琪, 任正某, 孙晓晨, 孙毅, 赵一伊. 基于联邦强化学习的电热综合能源系统能量管理策略[J]. 上海交通大学学报, 2024, 58(6): 904-915.

WANG Jinfeng, WANG Qi, REN Zhengmou, SUN Xiaochen, SUN Yi, ZHAO Yiyi. Energy Management Strategy of Integrated Electricity-Heat Energy System Based on Federated Reinforcement Learning[J]. Journal of Shanghai Jiao Tong University, 2024, 58(6): 904-915.

图/表 12

图1

图2

图3

表1

表2

表3

表4

图4

图5

图6

图7

表5

参考文献 29

[1]	苏慧玲, 杨世海, 陈铭明. 考虑能源效率的综合能源系统多目标优化调度[J]. 电力系统及其自动化学报, 2022, 34(2): 130-136.
	SU Huiling, YANG Shihai, CHEN Mingming. Multi-objective optimal scheduling of integrated energy system considering energy efficiency[J]. Proceedings of the CSU-EPSA, 2022, 34(2): 130-136.
[2]	董文杰, 田廓, 陈云斐, 等. 能源互联网下基于博弈与证据理论的综合能源系统评价方法研究[J]. 智慧电力, 2020, 48(7): 73-80.
	DONG Wenjie, TIAN Kuo, CHEN Yunfei, et al. Evaluation method of comprehensive energy system based on game theory & evidence theory under energy Internet[J]. Smart Power, 2020, 48(7): 73-80.
[3]	朱志芳, 许苑, 岑海凤, 等. 考虑需求侧响应的园区综合能源系统优化配置[J]. 智慧电力, 2022, 50(1): 37-44.
	ZHU Zhifang, XU Yuan, CEN Haifeng, et al. Optimal configuration of park-level integrated energy system considering demand response[J]. Smart Power, 2022, 50(1): 37-44.
[4]	李驰宇, 高红均, 刘友波, 等. 多园区微网优化共享运行策略[J]. 电力自动化设备, 2020, 40(3): 29-36.
	LI Chiyu, GAO Hongjun, LIU Youbo, et al. Optimal sharing operation strategy for multi park-level microgrid[J]. Electric Power Automation Equipment, 2020, 40(3): 29-36.
[5]	熊珞琳, 毛帅, 唐漾, 等. 基于强化学习的综合能源系统管理综述[J]. 自动化学报, 2021, 47(10): 2321-2340.
	XIONG Luolin, MAO Shuai, TANG Yang, et al. Reinforcement learning based integrated energy system management: A survey[J]. Acta Automatica Sinica, 2021, 47(10): 2321-2340.
[6]	CARLI R, DOTOLI M. Decentralized control for residential energy management of a smart users microgrid with renewable energy exchange[J]. CAA Journal of Automatica Sinica, 2019, 6(3): 641-656.
[7]	FARROKHIFAR M, AGHDAM F H, ALAHYARI A, et al. Optimal energy management and sizing of renewable energy and battery systems in residential sectors via a stochastic MILP model[J]. Electric Power Systems Research, 2020, 187: 106483.
[8]	ALIPOUR M, ZARE K, ABAPOUR M. MINLP probabilistic scheduling model for demand response programs integrated energy hubs[J]. IEEE Transactions on Industrial Informatics, 2018, 14(1): 79-88.
[9]	MOSER A, MUSCHICK D, GÖLLES M, et al. A MILP-based modular energy management system for urban multi-energy systems: Performance and sensitivity analysis[J]. Applied Energy, 2020, 261: 114342.
[10]	杨家豪. 区域综合能源系统冷-热-电-气概率多能流计算[J]. 电网技术, 2019, 43(1): 74-82.
	YANG Jiahao. Calculation of cold-heat-electricity-gas probabilistic multi-energy flow in regional comprehensive energy system[J]. Power System Technology, 2019, 43(1): 74-82.
[11]	翟晶晶, 吴晓蓓, 傅质馨, 等. 考虑需求响应与光伏不确定性的综合能源系统鲁棒优化[J]. 中国电力, 2020, 53(8): 9-18.
	ZHAI Jingjing, WU Xiaobei, FU Zhixin, et al. Robust optimization of integrated energy systems considering demand response and photovoltaic uncertainty[J]. Electric Power, 2020, 53(8): 9-18.
[12]	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题[J]. 自动化学报, 2020, 46(7): 1301-1312.
	SUN Changyin, MU Chaoxu. Important scientific problems of multi-agent deep reinforcement learning[J]. Acta Automatica Sinica, 2020, 46(7): 1301-1312.
[13]	刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803.
	LIU Junfeng, CHEN Jianlong, WANG Xiaosheng, et al. Research on energy management and optimization strategy of micro-energy network based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794-3803.
[14]	乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-832.
	QIAO Ji, WANG Xinying, ZHANG Qing, et al. Optimal dispatching of electric-gas integrated energy system based on flexible actuator-judge deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-832.
[15]	ZHANG B, HU W H, LI J H, et al. Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: Deep reinforcement learning approach[J]. Energy Conversion & Management, 2020, 220: 113063
[16]	XI L, YU L, XU Y C, et al. A novel multi-agent DDQN-AD method-based distributed strategy for automatic generation control of integrated energy systems[J]. IEEE Transactions on Sustainable Energy, 2020, 11(4): 2417-2426.
[17]	陈明昊, 孙毅, 胡亚杰, 等. 基于纵向联邦强化学习的居民社区综合能源系统协同训练与优化管理方法[J]. 中国电机工程学报, 2022, 42(15): 5535-5549.
	CHEN Minghao, SUN Yi, HU Yajie, et al. The collaborative training and management-optimized method for residential integrated energy system based on vertical federated reinforcement learning[J]. Proceedings of the CSEE, 2022, 42(15): 5535-5549.
[18]	SAPUTRA Y M, HOANG D T, NGUYEN D N, et al. Energy demand prediction with federated learning for electric vehicle networks[C]//2019 IEEE Global Communications Conference. Waikoloa, USA: IEEE, 2019: 1-6.
[19]	LEE S, CHOI D H. Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources[J]. IEEE Transactions on Industrial Informatics, 2022, 18(1): 488-497.
[20]	周长城, 马溪原, 郭晓斌, 等. 基于主从博弈的工业园区综合能源系统互动优化运行方法[J]. 电力系统自动化, 2019, 43(7): 74-80.
	ZHOU Changcheng, MA Xiyuan, GUO Xiaobin, et al. Leader-follower game based optimized operation method for interaction of integrated energy system in industrial park[J]. Automation of Electric Power Systems, 2019, 43(7): 74-80.
[21]	吕佳炜, 张沈习, 程浩忠. 计及热惯性和运行策略的综合能源系统可靠性评估方法[J]. 电力系统自动化, 2018, 42(20): 9-16.
	LYU Jiawei, ZHANG Shenxi, CHENG Haozhong. Reliability evaluation of integrated energy system considering thermal inertia and operation strategy[J]. Automation of Electric Power Systems, 2018, 42(20): 9-16.
[22]	白庆林, 张培山, 张林江, 等. 城市燃气SCADA及信息管理系统设计[J]. 自动化与仪表, 2009, 24(4): 47-50.
	BAI Qinglin, ZHANG Peishan, ZHANG Linjiang, et al. Design of information management system for urban natural gas SCADA system[J]. Automation & Instrumentation, 2009, 24(4): 47-50.
[23]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[24]	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California, USA: ACM, 2017: 6382-6393.
[25]	KOBAYASHI T, ILBOUDO W E L. T-soft update of target network for deep reinforcement learning[J]. Neural Networks, 2021, 136: 63-71. doi: 10.1016/j.neunet.2020.12.023 pmid: 33450653
[26]	MCKENNA E, THOMSON M. High-resolution stochastic integrated thermal-electrical domestic demand model[J]. Applied Energy, 2016, 165: 445-461.
[27]	THOMAS D, D’HOOP G, DEBLECKER O, et al. An integrated tool for optimal energy scheduling and power quality improvement of a microgrid under multiple demand response schemes[J]. Applied Energy, 2020, 260: 114314.
[28]	AYYADI S, BILIL H, MAAROUFI M. Optimal charging of Electric Vehicles in residential area[J]. Sustainable Energy, Grids & Networks, 2019, 19: 100240.
[29]	SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. 2nd ed. Cambridge, Massachusetts: The MIT Press, 2018.

供能子系统	所包含设备
MG₁	风力发电机、光伏、CHP机组、电、热储能系统、燃气锅炉、电热锅炉及电、热负荷
MG₂	风力发电机、光伏、CHP机组、电、热储能系统、燃气锅炉及电、热负荷
MG₃	风力发电机、光伏、电、热储能系统、燃气锅炉、电热锅炉及电、热负荷

设备	P^min, H^min/MW	P^max, H^max/MW
CHP机组	0	1.2
电热锅炉	0	1.4
燃气锅炉	0	1.2
蓄电池	-0.2	0.2

参数	数值	参数	数值
η_CHP	0.32	η_EB	0.94
β	1.25	η_GB	0.82
η_HE	0.73	C_Ncurt	0.20

时段	购电电价/ [元·(kW·h)^-1]	售电电价/ [元·(kW·h)^-1]
峰时段	0.922	0.500
平时段	0.540	0.200
谷时段	0.382	0

优化方法	成本/元			有效训练时长/s
优化方法	MG₁	MG₂	MG₃	有效训练时长/s
传统方法	19342.00	16786.00	49560.00	14228.00
MADDPG	18424.00	16629.00	47892.00	10634.20
F-DDPG	12973.02	13845.09	41469.00	6121.80