基于混合动作安全强化学习的汽油调合调度优化

Gasoline Blending Scheduling Optimization Based on Hybrid-Action Safe Reinforcement Learning

摘要

针对汽油调合调度中传统优化方法难以兼顾实时性、安全性与模型适配性的问题，提出了一种基于混合动作安全强化学习的调合调度优化方法，旨在实现随机扰动下的高效安全决策。该方法采用多通道前向传播（Multi-pass）结构消除混合动作评估中的虚假梯度，并结合比例-积分-微分（PID）调节的拉格朗日乘子机制处理质量与库存约束。仿真结果表明，所提方法在随机扰动下实现了产品质量与库存约束零违背，单步决策时间约 5 ms。尽管平均收益较确定性最优解低 2.5%，但该方法在复杂扰动环境下仍表现出优越的鲁棒性与实时应用潜力。

Abstract

A safe reinforcement learning method is proposed for gasoline blending scheduling under random disturbances in the properties and flow rates of direct-feed component oils. The scheduling task requires the coordinated optimization of discrete scheduling decisions and continuous blending recipes under strict product quality and inventory constraints. To solve this problem, the scheduling process is formulated in a hybrid action space so that discrete product switching decisions and continuous recipe regulation can be handled in a unified framework. On this basis, a Multi-pass hybrid soft actor-critic algorithm (MPH-SAC) is developed for gasoline blending scheduling. The Multi-pass structure removes spurious gradients in hybrid action evaluation and improves the coordination between discrete scheduling decisions and continuous blending recipes. In addition, a proportional-integral-derivative (PID) regulated Lagrangian multiplier mechanism is introduced to dynamically enforce safety constraints while maintaining economic performance during learning. The proposed method is tested in a gasoline blending scheduling environment with uncertain direct-feed component oil conditions. Simulation results show that the proposed method achieves zero violations of product quality and inventory constraints under random disturbances. The average online decision time is about 5 ms, whereas the deterministic optimization model solved by Gurobi requires about 394 s to obtain a near-optimal solution. Although the average profit of the proposed method is about 2.5% lower than the deterministic optimum, it maintains strong robustness against disturbances in the properties and flow rates of direct-feed component oils and avoids the model mismatch caused by deterministic assumptions. These results demonstrate that the proposed method can effectively balance profitability, operational safety, and real-time decision requirements, and it provides a practical solution for real-time gasoline blending scheduling in complex and uncertain operating environments.

关键词

汽油调合调度 / 安全强化学习 / 混合动作空间 / 拉格朗日乘子 / 优化问题

Key words

gasoline blending scheduling / safe reinforcement learning / hybrid action space / Lagrange multiplier / optimization problem

参考文献

[1] CASTILLO-CASTILLO P A, MAHALEC V. Inventory pinch gasoline blend scheduling algorithm combining discrete-and continuous-time models[J].Computers & Chemical Engineering, 2016, 84:611-626.

[2] GLISMANN K, GRUHN G. Short-term scheduling and recipe optimization of blending processes[J].Computers & Chemical Engineering, 2001, 25(4/6):627-634.

[3] LI J, KARIMI I A. Scheduling gasoline blending operations from recipe determination to shipping using unit slots[J].Industrial & Engineering Chemistry Research, 2011, 50(15):9156-9174.

[4] CERDÁ J, PAUTASSO P C, CAFARO D C. Optimizing gasoline recipes and blending operations using nonlinear blend models[J].Industrial & Engineering Chemistry Research, 2016, 55(28):7782-7800.

[5] ZHU Z, YANG M, HE W, et al. A deep reinforcement learning approach to gasoline blending real-time optimization under uncertainty[J].Chinese Journal of Chemical Engineering, 2024, 71:183-192.

[6] TERRAZAS-MORENO S, GROSSMANN I E. A multiscale decomposition method for the optimal planning and scheduling of multi-site continuous multiproduct plants[J].Chemical Engineering Science, 2011, 66(19):4307-4318.

[7] CERDÁ J, PAUTASSO P C, CAFARO D C. Optimization approaches for efficient crude blending in large oil refineries[J].Industrial & Engineering Chemistry Research, 2018, 57(25):8484-8501.

[8] JOLY M, MORO L F L, PINTO J M. Planning and scheduling for petroleum refineries using mathematical programming[J].Brazilian Journal of Chemical Engineering, 2002, 19:207-228.

[9] FRANZOI R E, MENEZES B C, KELLY J D, et al. Large-scale optimization of nonconvex MINLP refinery scheduling[J].Computers & Chemical Engineering, 2024, 186:108678.

[10] CASTILLO P A C, MAHALEC V, KELLY J D. Inventory pinch algorithm for gasoline blend planning[J].AIChE Journal, 2013, 59(10):3748-3766.

[11] ZHAO X, WANG Y. Gasoline blending scheduling based on uncertainty[C]//2009 International Conference on Computational Intelligence and Natural Computing. Wuhan, China: IEEE, 2009:84-87.

[12] YANG Y. Optimal blending under general uncertainties: A chance-constrained programming approach[J].Computers & Chemical Engineering, 2023, 171:108170.

[13] DAI X, ZHAO L, LI Z, et al. A data-driven approach for crude oil scheduling optimization under product yield uncertainty[J].Chemical Engineering Science, 2021, 246:116971.

[14] 罗春鹏, 荣冈. 不确定条件下汽油调和调度的鲁棒优化模型 [J].石油学报 (石油加工), 2009, 25(3):391-400.

[15] HANEVELD K, VAN DER VLERK M H. Stochastic Programming [M]. New York: Springer New York,2020.

[16] SKORDILIS E, MOGHADDASS R. A deep reinforcement learning approach for real-time sensor-driven decision making and predictive analytics[J].Computers & Industrial Engineering, 2020, 147:106600.

[17] 郎峻, 殷晓明, 顾幸生. 基于强化学习的正弦优化算法求解能耗分布式流水车间节能调度问题 [J].华东理工大学学报 (自然科学版), 2025, 51(5):623-632.

[18] 蔡林逸, 冯翔, 虞慧群. 基于组稀疏优化的强化学习稀疏表征 [J].华东理工大学学报 (自然科学版), 2024, 50(6):913-919.

[19] 许毅, 冯翔, 虞慧群. 基于强化学习和角度惩罚距离的冰晶连续优化算法 [J].华东理工大学学报 (自然科学版), 2021, 47(1):89-100.

[20] MASSON W, RANCHOD P, KONIDARIS G. Reinforcement learning with parameterized actions[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Arizona, USA: AAAI Press, 2016:1934-1940.

[21] KANERVISTO A, SCHELLER C, HAUTAMÄKI V. Action space shaping in deep reinforcement learning[C]//2020 IEEE conference on games (CoG). Osaka, Japan: IEEE, 2020:479-486.

[22] XIONG J, WANG Q, YANG Z, et al. Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space[EB/OL]. (2018-10-10). https://arxiv.org/abs/1810.06394.

[23] BESTER C J, JAMES S D, KONIDARIS G D. Multi-pass q-networks for deep reinforcement learning with parameterised action spaces[EB/OL]. (2019-5-10). https://arxiv.org/abs/1905.04388.

[24] WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//International conference on machine learning. New York, USA: PMLR, 2016:1995-2003.

[25] ZHOU H, LIN Z, LI J, et al. Revisiting discrete soft actor-critic[EB/OL]. (2022-9-22). https://arxiv.org/abs/2209.10081.

[26] HAARNOJA T, TANG H, ABBEEL P, et al. Reinforcement learning with deep energy-based policies[C]//International Conference on Machine Learning. Sydney, Australia: PMLR, 2017:1352-1361.

[27] HAUSKNECHT M, STONE P. Deep reinforcement learning in parameterized action space[EB/OL]. (2015-12-10). https://arxiv.org/abs/1511.04143.

[28] 王雪松, 王荣荣, 程玉虎. 安全强化学习综述 [J].自动化学报, 2023, 49(9):1813-1835.

[29] STOOKE A, ACHIAM J, ABBEEL P. Responsive safety in reinforcement learning by pid lagrangian methods[C]//International Conference on Machine Learning. Virtual Event: PMLR, 2020:9133-9143.

[30] XU Y, HU D, LIANG L, et al. Target entropy annealing for discrete soft actor-critic[EB/OL]. (2021-12-06). https://arxiv.org/abs/2112.02852.

基金资助

国家科技重大专项(2025ZD1608400)

国家自然科学基金重大项目(62394343)

中央高校基本科研业务费专项(222202617006)

高等学校学科创新引智计划(B17017)