网络化系统的广义策略迭代事件触发最优控制

Generalized Policy Iteration Event-triggered Optimal Control for Networked Systems

摘要

本文针对一类离散时间非线性网络化系统的最优控制问题,提出了一种基于广义策略迭代的自适应动态规划方法,并采用事件触发协议以降低系统通信负担.首先,设计了一种新型自适应事件触发协议,根据系统运行状态动态地调整触发阈值,并分析了事件触发下的系统的稳定性.其次,使用广义策略迭代近似求解时间触发下的哈密尔顿-雅可比-贝尔曼方程,兼顾了算法收敛速度与计算复杂度,并从理论上证明了该算法的有效性.然后,利用时间触发方程求解事件触发控制律,避免了重新构造事件触发方程带来的误差,并给出了数据经过事件触发协议后系统真实性能指标的上界.最后,通过仿真实例验证了本文方法的有效性.

Abstract

The optimal control problem for a class of discrete-time nonlinear networked systems is addressed,and an adaptive dynamic programming method based on generalized policy iteration is proposed,with an event-triggered protocol introduced to reduce the communication burden of the system.First,a novel adaptive event-triggered protocol is designed,in which the triggering threshold is dynamically adjusted according to the operating state of the system,and the stability of the system under event-triggered conditions is analyzed.Second,the Hamilton-Jacobi-Bellman equation under time-triggered conditions is approximated using generalized policy iteration,achieving a balance between algorithm convergence speed and computational complexity,and the effectiveness of the algorithm is theoretically proven.Then,the event-triggered control law is derived by utilizing the time-triggered equation,which avoids the modeling errors caused by reconstructing an event-triggered equation,and the upper bound of the actual performance index of the system with the event-triggered protocol is further obtained.Finally,the effectiveness of the proposed method is validated through a numerical simulation example.

关键词

自适应动态规划 / 网络化系统 / 广义策略迭代 / 事件触发

Key words

adaptive dynamic programming / networked systems / generalized policy iteration / event-triggered

[1] Qiu J B,Gao H J,Ding S X.Recent advances on fuzzy-model-based nonlinear networked control systems:a survey[J].IEEE Transactions on Industrial Electronics,2016,63(2):1207-1217.
[2] Hu C,Wang Z A,Bu X W,et al.Optimal tracking control for autonomous vehicle with prescribed performance via adaptive dynamic programming[J].IEEE Transactions on Intelligent Transportation Systems,2024,25(9):12437-12449.
[3] Wang L,Qi R Y,Jiang B.Adaptive fault-tolerant control for non-minimum phase hypersonic vehicles based on adaptive dynamic programming[J].Chinese Journal of Aeronautics,2024,37(3):290-311.
[4] Yuan L H,Wang L D,Zhang J X.Adaptive dynamic programming base on MMC device of a flexible high-altitude long endurance aircraft[J].Aerospace Science and Technology,2024,151:109305.doi:10.1016/j.ast.2024.109305.
[5] XIAO J,TANG Q P,LI J L,et al.Formation control method for multiple flight vehicles based on adaptive dynamic programming[J].Chinese Journal of Engineering,2025,47(5):1094-1102.
[6] Zhang Y H,Zou L,Liu Y,et al.A brief survey on nonlinear control using adaptive dynamic programming under engineering-oriented complexities[J].International Journal of Systems Science,2023,54(8):1855-1872.
[7] CHEN J,ZHANG F J,WANG K.Event triggered control of a class of nonlinear multi-agent based on adaptive dynamic programming[J].Computer Application and Software,2025,42(7):155-160.
[8] Yang Y,Fan X,Gao W N,et al.Event-triggered output feedback control for a class of nonlinear systems via disturbance observer and adaptive dynamic programming [J].IEEE Transactions on Fuzzy Systems,2023,31(9):3148-3160.
[9] Xie K D,Zheng Y W,Jiang Y,et al.Optimal dynamic output feedback control of unknown linear continuous-time systems by adaptive dynamic programming[J].Automatica,2024,163:111601.doi:10.1016/j.automatica.2024.111601.
[10] TAN X F,LI Y,LIU Y.Stochastic linear quadratic optimal tracking control with time-delays based on adaptive dynamic programming[J].Journal of Systems Science and Mathematical Sciences,2024,44(1):17-30.
[11] Yang X,Wei Q L.Adaptive dynamic programming for robust event-driven tracking control of nonlinear systems with asymmetric input constraints[J].IEEE Transactions on Cybernetics,2024,54(11):6333-6344.
[12] Wang D,Gao N,Liu D R,et al.Recent progress in reinforcement learning and adaptive dynamic programming for advanced control applications[J].IEEE/CAA Journal of Automatica Sinica,2024,11(1):18-36.
[13] Zhang H G,Zhang L L,Sun J Y,et al.Optimal control for unknown nonlinear system with semi-markovian jump parameters via adaptive dynamic programming[J].IEEE Transactions on Systems,Man,and Cybernetics:Systems,2024,54(10):6255-6264.
[14] Cai X J,Gao Q,Liu H,et al.Multiplayer hierarchical decision-making for discrete-time nonlinear networks of service via value iteration adaptive dynamic programming[J].International Journal of Robust and Nonlinear Control,2024,doi:10.1002/rnc.7653.
[15] Zhang H.An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems[J].Journal of Applied Mathematics and Computing,2023,69(3):2741-2760.
[16] Wei Q L,Yang Z S,Su H Z,et al.Online adaptive dynamic programming for optimal self-learning control of VTOL aircraft systems with disturbances[J].IEEE Transactions on Automation Science and Engineering,2024,21(1):343-352.
[17] WU Y D,TAO P,ZHANG Y R,et al.Stochastic distributed secondary control of micro-grid based on event triggered communication[J].Computer Applications Software,2025,42(7):96-106.
[18] Li C C,Zhao X D,Chen M,et al.Dynamic periodic event-triggered control for networked control systems under packet dropouts[J].IEEE Transactions on Automation Science and Engineering,2024,21(1):906-920.
[19] FAN Q Y,ZHANG N Z,TANG Y,et al.Adaptive reliable control of multi-agent systems based on dynamic event-triggered communication protocol[J].Acta Automatica Sinica,2024,50(5):924-936.
[20] Yang X,Wei Q L.Adaptive critic learning for constrained optimal event-triggered control with discounted cost[J].IEEE Transactions on Neural Networks and Learning Systems,2021,32(1):91-104.
[21] Shen M Q,Wang X M,Zhu S,et al.Data-driven event-triggered adaptive dynamic programming control for nonlinear systems with input saturation[J].IEEE Transactions on Cybernetics,2024,54(2):1178-1188.
[22] Han X M,Hao M N,Li P,et al.Quantized optimal output feedback control and optimal triggering signal co-design for unknown discrete-time nonlinear systems[J].Optimal Control Applications and Methods,2022,43(3):962-978.
[23] Zhang K,Zhang Z X,Xie X P,et al.An unknown multiplayer nonzero-sum game:prescribed-time dynamic event-triggered control via adaptive dynamic programming[J].IEEE Transactions on Automation Science and Engineering,2025,22:8317-8328,doi:10.1109/TASE.2024.3484412.
[24] Lu J W,Wei Q L,Zhou T M,et al.Event-triggered near-optimal control for unknown discrete-time nonlinear systems using parallel control[J].IEEE Transactions on Cybernetics,2023,53(3):1890-1904.
[25] Luo B,Yang Y,Liu D R,et al.Event-triggered optimal control with performance guarantees using adaptive dynamic programming[J].IEEE Transactions on Neural Networks and Learning Systems,2020,31(1):76-88.
[26] Wei Q L,Lu J W,Zhou T M,et al.Event-triggered near-optimal control of discrete-time constrained nonlinear systems with application to a boiler-turbine system[J].IEEE Transactions on Industrial Informatics,2022,18(6):3926-3935.
[27] Pan S Y,Zhang Y Z,He P,et al.A new multi-iteration adaptive dynamic programming method for smart grid systems[C]//8th International Conference on Automation,Control and Robots(ICACR),2024:117-121.
[28] Zhu J J,Zhao X Y,Li Y X.Trajectory tracking of intelligent vehicle systems based on sampled data and adaptive dynamic programming[C]//10th International Forum on Electrical Engineering and Automation(IFEEA),2023:990-996.
[29] Zhang C F,Zhang G S,Dong Q.Adaptive dynamic programming-based adaptive-gain sliding mode tracking control for fixed-wing unmanned aerial vehicle with disturbances[J].International Journal of Robust and Nonlinear Control,2023,33(2):1065-1097.
[30] Liu D R,Xue S,Zhao B,et al.Adaptive dynamic programming for control:a survey and recent advances[J].IEEE Transactions on Systems,Man,and Cybernetics:Systems,2020,51(1):142-160.
附中文参考文献:
[5] 肖 杰,汤清璞,李嘉乐,等.基于自适应动态规划的多飞行器编队控制方法[J].工程科学学报,2025,47(5):1094-1102.
[7] 陈 炯,张飞建,王 凯.基于自适应动态规划的一类非线性多智能体的事件触发控制[J].计算机应用与软件,2025,42(7):155-160.
[10] 谭旭峰,李 媛,刘 洋.基于自适应动态规划的随机时滞线性二次型最优跟踪控制[J].系统科学与数学,2024,44(1):17-30.
[17] 吴一敌,陶 鹏,张洋瑞,等.事件触发通信的微电网随机分布式二级控制[J].计算机应用与软件,2025,42(7):96-106.
[19] 范泉涌,张乃宗,唐 勇,等.基于动态事件触发通信协议的多智能体系统自适应可靠控制[J].自动化学报,2024,50(5):924-936

基金资助

黑龙江省优秀青年基金项目(YQ2023F002)资助;国家自然科学基金项目(52077075)资助.