基于深度强化学习算法的水光互补优化调度研究

Research on water-light complementary optimal scheduling based on deep reinforcement learning algorithm

摘要

【目的】水光互补优化调度中光伏出力具有波动性、随机性、间歇性等特点，其求解空间通常较高维、复杂且为连续空间，水光互补优化调度问题中涉及的多种连续控制决策问题。【方法】深度强化学习算法中的深度确定性策略梯度(DDPG)算法适合处理求解空间连续、复杂的问题，首先对水光互补问题进行强化学习建模，并基于水光互补机理考虑需调、可调概念设置环境、动作、奖励函数与惩罚函数，采用DDPG算法进行优化。对比分析只使用初始DDPG算法和遗传算法的优化结果，讨论模型的适用性和有效性。以澜沧江上游大型水光互补基地为例，设置三个梯级水电站配置方案，三种水文代表年，开展实例研究。【结果】(1)采用DDPG算法运行速度较快，在考虑需调、可调水光互补机理建模后光伏消纳电量达到129.93亿kWh,为三种模型最高。(2)来水越枯，光伏消纳能力越强；梯级水电站装机862万kW时，平水年到枯水年光伏消纳能力只提高1%,此时能够最大限度利用水电系统的互补能力。(3)在丰水期的光伏消纳能力较低，三个方案光伏消纳率分别为77.43%、79.85%、89.39。【结论】深度强化学习算法在水光互补优化调度中展现出快速收敛的优势，将“需调”和“可调”机理融入强化学习建模能够显著提升光伏消纳效果，实现更优的资源利用，有效提升了水光互补系统的运行效率与光伏电量消纳能力。该方法在清洁能源基地的容量配置和运行调度领域有较好效果，为后续清洁能源系统的扩展应用奠定了理论与实践基础。

Abstract

[Objective] Photovoltaic output in water-light complementary optimal scheduling is characterized by volatility, randomness, and intermittency. Its solution space is typically high-dimensional, complex, and continuous. A variety of continuous control decision-making problems are involved in water-light complementary optimal scheduling. [Methods] The Deep Deterministic Policy Gradient(DDPG) algorithm in the deep reinforcement learning algorithm was suitable for solving continuous and complex problems in the solution space. The water-light complementary problem was modeled using reinforcement learning. Based on the water-light complementary mechanism, the concepts of “demand for adjustment” and “capacity for adjustment” were considered to set up the environment, actions, reward function, and penalty function. The DDPG algorithm was then used for optimization. The applicability and effectiveness of the model were assessed by comparing and analyzing the optimization result using only the initial DDPG algorithm and those using the genetic algorithm. Taking the large-scale water-light complementary base in the upper reaches of the Lancang River as an example, three cascade hydropower station configuration schemes and three representative hydrological years were set up for a case study. [Results] The analysis indicated that:(1) the DDPG algorithm performed faster. After considering the mechanisms of “demand for adjustment” and “capacity for adjustment”, the photovoltaic power consumption reached 12.993 billion kWh, which was the highest among the three models.(2) The lower the water inflow, the stronger the photovoltaic consumption capacity was. When the installed capacity of cascade hydropower stations was 8.62 million kW, the photovoltaic consumption capacity only increased by 1% from normal year to dry year. At this time, the complementary capacity of hydropower system could be maximized.(3) The photovoltaic consumption capacity was relatively low during the wet season, and the photovoltaic consumption rates of the three schemes were 77.43%, 79.85%, and 89.39%, respectively. [Conclusion] The deep reinforcement learning algorithm demonstrates the advantage of rapid convergence in the water-light complementary optimal scheduling. Integrating mechanisms of “demand for adjustment” and “capacity for adjustment” into reinforcement learning modeling can significantly enhance the photovoltaic consumption efficiency, achieve better resource utilization, and effectively improve the operation efficiency of the water-light complementary system and the photovoltaic power consumption capacity. This method shows promising result in the capacity configuration and operation scheduling of clean energy base, providing a theoretical and practical foundation for the future expansion and application of clean energy system.

关键词

水光互补 / 强化学习 / DDPG / 优化调度 / 影响因素 / 水电站

Key words

water-light complementary / reinforcement learning / DDPG / optimal scheduling / influencing factors / hydropower stations

基金资助

国家自然科学基金项目(52179012)

中国华能集团科技项目“澜沧江西藏段千万千瓦清洁能源基地建设水光互补关键技术研究”(HNKJ20-H20)