New Type Power System and the Integrated Energy

Renewable Energy Consumption Strategies of Power System Integrated with Electric Vehicle Clusters Based on Load Alignment and Deep Reinforcement Learning

  • LIU Yanhang ,
  • QIAO Ruyu ,
  • LIANG Nan ,
  • CHEN Yu ,
  • YU Kai ,
  • WU Hanxiao
Expand
  • 1 Power Marketing Service and Operation Management Branch, Inner Mongolia Power (Group) Co., Ltd., Hohhot 010011, China
    2 Key Laboratory of Control of Power Transmission and Conversion of the Ministry of Education, Shanghai Jiao Tong University, Shanghai 200240, China

Revised date: 2023-11-20

  Accepted date: 2023-11-30

  Online published: 2023-12-12

Abstract

As China accelerates the construction of power systems with renewable energy as the mainstay, the large-scale integration of renewables has led to prominent issues such as wind and light curtailment. To improve the utilization of new energy consumption in power systems, this paper proposes a novel renewable energy consumption method based on load alignment and deep reinforcement learning. First, it proposes a node load line formation model based on linearized power flow calculations, which can guide adjustable loads to shift the electricity consumption period, thereby promoting the improvement of new energy consumption. Unlike the direct current (DC) power flow model, the proposed alternating current (AC) model accounts for voltage constraints and other related constraints of the power system. Compared with other AC power flow models, this model linearizes all nonlinear constraints and has lower computational costs. Then, this paper constructs a market framework for load alignment mechanism. The framework involves three main entities: independent system operators, regional power grid sellers, and electric vehicle adjustable load aggregators. It also explores the solution for load alignment incentive prices using electric vehicle clusters as adjustable loads. As the solution of the load benchmark incentive price involves a master-slave game between three entities, conventional mathematical analysis methods face high complexity. Therefore, it employs deep reinforcement learning algorithm to solve the problem. The deep reinforcement learning algorithm takes the marginal electricity price of each node as state space, the load benchmark incentive price as action space, and the cost of regional power grid sellers as feedback. The agent can find the load line incentive price that maximizes the benefits of regional power grid sellers after continuous training. Finally, the example analysis shows that the load alignment mechanism not only effectively promotes the improvement of new energy consumption level, but also enhances the interests of independent system operators, regional power grid sellers, and electric vehicle aggregators. The results further confirm that the deep reinforcement learning algorithm maximizes the benefits of regional power grid sellers.

Cite this article

LIU Yanhang , QIAO Ruyu , LIANG Nan , CHEN Yu , YU Kai , WU Hanxiao . Renewable Energy Consumption Strategies of Power System Integrated with Electric Vehicle Clusters Based on Load Alignment and Deep Reinforcement Learning[J]. Journal of Shanghai Jiaotong University, 2025 , 59(10) : 1464 -1475 . DOI: 10.16183/j.cnki.jsjtu.2023.529

References

[1] 张英杰. 构建以新能源为主体的新型电力系统的发展路径研究[J]. 电工技术, 2022(18): 172-174.
  ZHANG Yingjie. Research on the development path of building a new electric power system based on new energy sources[J]. Electric Engineering, 2022(18): 172-174.
[2] 江婷, 邓晖, 陆承宇, 等. 电能量和旋转备用市场下电-热综合能源系统低碳优化运行[J]. 上海交通大学学报, 2021, 55(12): 1650-1662.
  JIANG Ting, DENG Hui, LU Chengyu, et al. Low-carbon optimal operation of an integrated electricity-heat energy system in electric energy and spinning reserve market[J]. Journal of Shanghai Jiao Tong University, 2021, 55(12): 1650-1662.
[3] 李志伟, 赵雨泽, 吴培. 基于制氢设备精细建模的综合能源系统绿氢蓝氢协调低碳优化策略[J]. 电网技术, 2024, 48(6): 2317-2326.
  LI Zhiwei, ZHAO Yuze, WU Pei, et al. Low-carbon dispatching strategy of integrated energy system with coordination of green hydrogen and blue hydrogen based on fine modeling of hydrogen production equipment[J]. Power System Technology, 2024, 48(6): 2317-2326.
[4] 岑彬. “双碳”背景下可再生能源发展中“弃风弃光”的问题及消纳措施[J]. 中阿科技论坛(中英文), 2022(10): 60-63.
  CEN Bin. Study on the “forced abandonment of wind and light” in the development of renewable energy under the background of “dual carbon” and its mitigation measures[J]. China-Arab States Science & Technology Forum, 2022(10): 60-63.
[5] 许高秀, 王旭, 邓晖, 等. 考虑调频需求及风光出力不确定性的储能系统参与能量-调频市场运行策略[J]. 电网技术, 2023, 47(6): 2317-2330.
  XU Gaoxiu, WANG Xu, DENG Hui, et al. Optimal operation strategy of energy storage system’s participation in energy and regulation market considering uncertainties of regulation requirements and wind-photovoltaic output[J]. Power System Technology, 2023, 47(6): 2317-2330.
[6] 李艳梅, 任恒君, 张致远, 等. 考虑储能系统调度与风电消纳的峰谷分时电价优化模型研究[J]. 电网技术, 2022, 46(11): 4141-4149.
  LI Yanmei, REN Hengjun, ZHANG Zhiyuan, et al. Optimization model of peak-valley time-of-use electricity prices considering energy storage system dispatching and wind power consumption[J]. Power System Technology, 2022, 46(11): 4141-4149.
[7] 尚文强, 李广磊, 丁月明, 等. 考虑源荷不确定性和新能源消纳的综合能源系统协同调度方法[J]. 电网技术, 2024, 48(2): 517-532.
  SHANG Wenqiang, LI Guanglei, DING Yueming, et al. A collaborative scheduling method for integrated energy system considering the uncertainty of source load and the absorption of new energy[J]. Power System Technology. 2024, 48(2): 517-532.
[8] 赵雅雪, 王旭, 蒋传文, 等. 基于最大信息系数相关性分析和改进多层级门控LSTM的短期电价预测方法[J]. 中国电机工程学报, 2021, 41(1): 135-146.
  ZHAO Yaxue, WANG Xu, JIANG Chuanwen, et al. A novel short-term electricity price forecasting method based on correlation analysis with the maximal information coefficient and modified multi-hierachy gated LSTM[J]. Proceedings of the CSEE, 2021, 41(1): 135-146.
[9] 范帅, 郏琨琪, 王芬, 等. 基于负荷准线的大规模需求响应[J]. 电力系统自动化, 2020, 44(15): 19-27.
  FAN Shuai, JIA Kunqi, WANG Fen, et al. Large-scale demand response based on customer directrix load[J]. Automation of Electric Power Systems, 2020, 44(15): 19-27.
[10] 刘春阳, 李康平, 纪陵, 等. 基于聚类-估计联动的需求响应集群基线负荷估计方法[J]. 电力系统自动化, 2023, 47(2): 79-87.
  LIU Chunyang, LI Kangping, JI Ling, et al. Clustering-estimation linkage based estimation method for aggregated baseline loads of demand response[J]. Automation of Electric Power Systems, 2023, 47(2): 79-87.
[11] 范帅, 危怡涵, 何光宇, 等. 面向新型电力系统的需求响应机制探讨[J]. 电力系统自动化, 2022, 46(7): 1-12.
  FAN Shuai, WEI Yihan, HE Guangyu, et al. Discussion on demand response mechanism for new power systems[J]. Automation of Electric Power Systems, 2022, 46(7): 1-12.
[12] 孟琰, 肖居承, 洪居华, 等. 计及需求响应不确定性的节点负荷准线:概念与模型[J]. 电力系统自动化, 2023, 47(13): 28-39.
  MENG Yan, XIAO Jucheng, HONG Juhua, et al. Nodal customer directrix load considering demand response uncertainty: Concept and model[J]. Automation of Electric Power Systems, 2023, 47(13): 28-39.
[13] 徐博强, 张沛超, 何光宇, 等. 基于主从博弈的热水器集群的负荷准线控制方法[J]. 中国电机工程学报, 2022, 42(21): 7785-7797.
  XU Boqiang, ZHANG Peichao, HE Guangyu, et al. Stackelberg game based control method for water heater cluster using customer directrix line[J]. Proceedings of the CSEE, 2022, 42(21): 7785-7797.
[14] WANG X, SHAHIDEHPOUR M, JIANG C W, et al. Coordinated planning strategy for electric vehicle charging stations and coupled traffic-electric networks[J]. IEEE Transactions on Power Systems, 2019, 34(1): 268-279.
[15] LI K, SHAO C C, ZHANG H C, et al. Strategic pricing of electric vehicle charging service providers in coupled power-transportation networks[J]. IEEE Transactions on Smart Grid, 2023, 14(3): 2189-2201.
[16] YANG Z F, ZHONG H W, BOSE A, et al. A linearized OPF model with reactive power and voltage magnitude: A pathway to improve the MW-only DC OPF[J]. IEEE Transactions on Power Systems, 2018, 33(2): 1734-1745.
[17] YANG Z F, ZHONG H W, BOSE A, et al. Optimal power flow in AC-DC grids with discrete control devices[J]. IEEE Transactions on Power Systems, 2018, 33(2): 1461-1472.
[18] 周士超, 刘晓林, 熊展, 等. 考虑韧性提升的交直流配电网线路加固和储能配置策略[J]. 上海交通大学学报, 2021, 55(12): 1619-1630.
  ZHOU Shichao, LIU Xiaolin, XIONG Zhan, et al. Line hardening and energy storage system configuration strategies for resilience enhancement of a hybrid AC-DC distribution system[J]. Journal of Shanghai Jiao Tong University, 2021, 55(12): 1619-1630.
[19] GONG K, WANG X, JIANG C W, et al. Security-constrained optimal sizing and siting of BESS in hybrid AC/DC microgrid considering post-contingency corrective rescheduling[J]. IEEE Transactions on Sustainable Energy, 2021, 12(4): 2110-2122.
[20] WANG X, SHAHIDEHPOUR M, JIANG C W, et al. Resilience enhancement strategies for power distribution network coupled with urban transportation system[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 4068-4079.
[21] 胡维昊, 曹迪, 黄琦, 等. 深度强化学习在配电网优化运行中的应用[J]. 电力系统自动化, 2023, 47(14): 174-191.
  HU Weihao, CAO Di, HUANG Qi, et al. Application of deep reinforcement learning in optimal operation of distribution network[J]. Automation of Electric Power Systems, 2023, 47(14): 174-191.
[22] DUAN J J, YI Z H, SHI D, et al. Reinforcement-learning-based optimal control of hybrid energy storage systems in hybrid AC-DC microgrids[J]. IEEE Transactions on Industrial Informatics, 2019, 15(9): 5355-5364.
[23] BUI V H, HUSSAIN A, KIM H M. Double deep Q-learning-based distributed operation of battery energy storage system considering uncertainties[J]. IEEE Transactions on Smart Grid, 2020, 11(1): 457-469.
[24] WANG S Y, DU L, FAN X Y, et al. Deep reinforcement scheduling of energy storage systems for real-time voltage regulation in unbalanced LV networks with high PV penetration[J]. IEEE Transactions on Sustainable Energy, 2021, 12(4): 2342-2352.
[25] 张继行, 张一, 王旭, 等. 基于多代理强化学习的多新型市场主体虚拟电厂博弈竞价及效益分配策略[J]. 电网技术, 2024, 48(5): 1980-1991.
  ZHANG Jihang, ZHANG Yi, WANG Xu, et al. Game bidding and benefit allocation strategy for virtual power plants with multiple new market entities based on multi-agent reinforcement learning[J]. Power System Technology, 2024, 48(5): 1980-1991.
[26] LIANG Y C, GUO C L, DING Z H, et al. Agent-based modeling in electricity market using deep deterministic policy gradient algorithm[J]. IEEE Transactions on Power Systems, 2020, 35(6): 4180-4192.
[27] 徐业琰, 姚良忠, 廖思阳, 等. 基于多智能体Actor-Double-Critic深度强化学习的源-网-荷-储实时优化调度方法研究[J/OL]. 中国电机工程学报. https://doi.org/10.13334/j.0258-8013.pcsee.231054.
  XU Yeyan, YAO Liangzhong, LIAO Siyang, et al. Studies on real-time optimal dispatch method of source-grid-load-storage based on multi-agent actor-double-critic deep reinforcement learning[J/OL]. Proceedings of the CSEE. https://doi.org/10.13334/j.0258-8013.pcsee.231054.
[28] LIU D N, GAO Y, WANG W Y, et al. Research on bidding strategy of thermal power companies in electricity market based on multi-agent deep deterministic policy gradient[J]. IEEE Access, 2021, 9: 81750-81764.
[29] 刘飞宇, 王吉文, 王正风, 等. 基于两阶段深度强化学习算法的多智能体自由合谋竞价机理研究[J]. 中国电机工程学报, 2024, 44(12): 4626-4639.
  LIU Feiyu, WANG Jiwen, WANG Zhengfeng, et al. Study on free joint bidding mechanism in multi-agent environment based on two-stage deep reinforcement learning algorithm[J]. Proceedings of the CSEE, 2024, 44(12): 4626-4639.
Outlines

/