线性二次随机系统的完全无模型 Ｑ——学习算法

A Q-learning Algorithm for Linear Quadratic Stochastic System with Completely Unknown Dynamics

摘要

本文提出了一种新型的无模型 Q - 学习算法，用于求解连续时间线性二次随机系统的自适应最优控制问题。基于由 Kleinman 算法衍生出的随机系统策略迭代算法，本文构建了一个由 Hamiltonian 函数和最优值函数组成的 Q 函数作为性能指标，开发了一种在没有任何模型信息的情况下，依据行动者 / 评论家结构迭代 Q 函数求得最优策略的数据驱动方法。此外，还保证了闭环稳定性和收敛性。最后，通过一个数值算例验证了该算法的有效性和可行性

Abstract

In this paper, a novel model-free Q-learning algorithm is proposed to solve the adaptive optimal control problem for continuous-time linear quadratic stochastic system. Based on the policy iterative algorithm derived from Kleinman’s algorithm for stochastic system, a Q function consisting of Hamiltonian and the optimal cost function has been selected as the performance index. Then, a data-driven strategy is developed to obtain the optimal policy on the basis of Q function under the actor/critic approximator structure without any knowledge of the model. Furthermore, the closed-loop stability and convergence have been guaranteed. Finally, a numerical example has been employed to verify the effectiveness and feasibility of the algorithm.

关键词

Q 函数 / 数据驱动 / 行动者 / 评论家 / 随机控制 / 无模型

Key words

Q function / data driven / actor/critic / stochastic control / model-free

参考文献

[1] Vamvoudakis K G. Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach[J]. Systems & Control Letters, 2017, 100: 14–20.

[2] Possieri C, Sassano M. Q-learning for continuous-time linear systems: A data driven implementation of the Kleinman algorithm[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52(10): 6487–6497.

[3] Pang B, Jiang Z P. Reinforcement learning for adaptive optimal stationary control of linear stochastic systems[J]. IEEE Transactions on Automatic Control, 2023, 68(4): 2383–2390.

[4] Zhang H. An adaptive dynamic programming-based algorithm for infinite-horizon linear quadratic stochastic optimal control problems[J]. Journal of Applied Mathematics and Computing, 2023, 69(3): 2741–2760.

[5] Kleinman D. On an iterative technique for Riccati equation computations[J]. IEEE Transactions on Automatic Control, 1968, 13(1): 114–115.

[6] Ni Y H, Fang H T. Policy iteration algorithm for singular controlled diffusion processes[J]. SIAM Journal on Control and Optimization, 2013, 51(5): 3844–3862.

[7] Rami M A, Zhou X Y. Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls[J]. IEEE Transactions on Automatic Control, 2000, 45(6): 1131–1143.

基金资助

江苏省自然科学基金资助项目(BK20191386)