基于双行动者双评论家算法的间歇过程优化控制

Optimal Control of Batch Process Based on Double Actors and Double Critics Algorithm

摘要

在使用深度强化学习 (Deep Reinforcement Learning, DRL) 方法对间歇过程进行优化控制时，存在样本学习效率低、训练时间长和训练不稳定的问题。针对这些问题，提出了双行动者正则化双评论家 (Double Actors and Regularized Double Critics, DARDC) 算法。算法通过并行双行动者网络增强智能体在复杂状态空间的探索能力，借助正则化项约束双评论家网络，有效抑制价值函数估计偏差，同时引入优先经验回放机制 (Prioritized Experience Replay, PER) 加速训练收敛。为保障智能体输出的控制量合理，在动作输出中嵌入专家经验规则。在青霉素发酵过程的实验中，所提算法的性能优于传统深度强化学习算法，且与比例-积分-微分（PID）控制相比，关键控制指标青霉素平均产量提升 19.1%。结果表明，DARDC 算法具备一定的有效性与鲁棒性，为间歇过程优化控制提供了新方法。

Abstract

When using deep reinforcement learning methods to optimize the control of batch processes, there are problems such as low sample learning efficiency, long training time and unstable training. To solve these problems, a Double Actors and Regularized Double Critics (DARDC) algorithm is proposed. The algorithm enhances the agent's exploration ability in the complex state space through the parallel double actor network, effectively suppresses the value function estimation deviation by virtue of the regularization term constraint double critic network, and introduces the prioritized experience replay mechanism to accelerate the training convergence. Further, in order to ensure the rationality of control quantities output by the agent, expert experience rules are embedded into the action output. In the experiment of penicillin fermentation process, the performance of the proposed algorithm is better than the traditional deep reinforcement learning algorithm, and compared with PID control, the average penicillin yield of the key control index is increased by 19.1%. The results show that the DARDC algorithm has certain effectiveness and robustness, and provides a new method for the optimal control of batch process.

关键词

深度强化学习 / 间歇过程优化控制 / 双行动者 / 双评论家 / 优先经验回放

Key words

deep reinforcement learning / optimal control of batch process / double actors / double critics / prioritized experience replay

参考文献

[1] SITTIG W. The present state of fermentation reactors[J].Journal of Chemical Technology and Biotechnology, 1982, 32(1):47-58.

[2] KUMAR A, UDUGAMA I A, GARGALO C L, et al. Why is batch processing still dominating the biologics landscape? Towards an integrated continuous bioprocessing alternative[J].Processes, 2020, 8(12):1641.

[3] GOLDRICK S, DURAN—VILLALOBOS C A, JANKAUSKAS K, et al. Modern day monitoring and control challenges outlined on an industrial—scale benchmark fermentation process[J].Computers & Chemical Engineering, 2019, 130:106471.

[4] DAEMMRICH A. Synthesis by microbes or chemists? Pharmaceutical research and manufacturing in the antibiotic era[J].History and Technology, 2009, 25(3):237-256.

[5] LI D, ZHU F, WANG X, et al. Multi—objective reinforcement learning for fed—batch fermentation process control[J].Journal of Process Control, 2022, 115:89-99.

[6] OH T H, PARK H M, KIM J W, et al.Integration of reinforcement learning and model predictive control to optimize semi—batch bioreactor[J].AIChE Journal, 2022, 68(6):e17658.

[7] MORARI M, LEE J H. Model predictive control: Past, present and future[J].Computers & Chemical Engineering, 1999, 23(4/5):667-682.

[8] HENSON M A. Nonlinear model predictive control: Current status and future directions[J].Computers & Chemical Engineering, 1998, 23:187-202.

[9] LOPEZ—MONTERO E B, WAN J, MARJANOVIC O. Trajectory tracking of batch product quality using intermittent measurements and moving window estimation[J].Journal of Process Control, 2015, 25:115-128.

[10] JANG H, LEE J H, BIEGLER L T. A robust NMPC scheme for semi—batch polymerization reactors[J].IFAC—PapersOnLine, 2016, 49(7):37-42.

[11] HONG M S, BRAATZ R D. Mechanistic modeling and parameter—adaptive nonlinear model predictive control of a microbioreactor[J].Computers & Chemical Engineering, 2021, 147:107255.

[12] SHAKYA A K, PILLAI G, CHAKRABARTY S. Reinforcement learning algorithms: A brief survey[J].Expert Systems with Applications, 2023, 231:120495.

[13] DOGRU O, XIE J, PRAKASH O, et al. Reinforcement learning in process industries: Review and perspective[J].IEEE/CAA Journal of Automatica Sinica, 2024, 11(2):283-300.

[14] NIAN R, LIU J, HUANG B. A review on reinforcement learning: Introduction and applications in industrial process control[J].Computers & Chemical Engineering, 2020, 139:106886.

[15] KIM J W, PARK B J, OH T H, et al. Model—based reinforcement learning and predictive control for two—stage optimal control of fed—batch bioreactor[J].Computers & Chemical Engineering, 2021, 154:107465.

[16] 邓绍斌, 朱军, 周晓锋, 等. 基于局部策略交互探索的深度确定性策略梯度的工业过程控制方法 [J].计算机应用, 2022, 42(5):1642-1648.

[17] LI H, QIU T, YOU F. AI—based optimal control of fed—batch biopharmaceutical process leveraging deep reinforcement learning[J].Chemical Engineering Science, 2024, 292:119990.

[18] FUJIMOTO S, HOOF H, MEGER D. Addressing function approximation error in actor—critic methods[C]//Proceedings of the 35th International Conference on Machine Learning. [s.l.]: PMLR,2018:1587-1596.

[19] JOSHI T, MAKKER S, KODAMANA H, et al. Twin actor twin delayed deep deterministic policy gradient (TATD3) learning for batch process control[J].Computers & Chemical Engineering, 2021, 155:107527.

[20] 马军伟, 徐琛, 陶洪峰, 等. 基于双行动者深度确定性策略梯度算法的间歇过程控制 [J].信息与控制, 2023, 52(6):773-783.

[21] ZHANG Y, ZHANG K, HU T, et al. Research on intelligent monitoring and concentration prediction for penicillin fermentation process[J].Biotechnology and Bioengineering. 2025, 122(3):570-578.

[22] LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[EB/OL]. (2015—09—09). https://arxiv.org/abs/1509.02971.

[23] KANG C, RONG C, REN W, et al. Deep deterministic policy gradient based on double network prioritized experience replay[J].IEEE Access, 2021, 9:60296-60308.

[24] LYU J, MA X, YAN J, et al. Efficient continuous control with double actors and regularized critics[J].Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(7):7655-7663.

[25] BONEY R, KANNALA J, ILIN A. Regularizing model—based planning with energy—based models[C]//Proceedings of the Conference on Robot Learning. [s.l.]: PMLR,2020:182-191.

[26] SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL]. (2015—11—18). https://arxiv.org/abs/1511.05952.

[27] GOLDRICK S, ŞTEFAN A, LOVETT D, et al. The development of an industrial—scale fed—batch fermentation simulation[J].Journal of Biotechnology, 2015, 193:70-82.

[28] HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor—critic: Off—policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of the 35th International Conference on Machine Learning. [s.l.]: PMLR,2018:1861-1870.

[29] ZHANG M, WANG X, DECARDI—NELSON B, et al. Smpl: Simulated industrial manufacturing and process control learning environments[J].Advances in Neural Information Processing Systems, 2022, 35:26631-26644.