Aiming at the problem of high overhead when applying reinforcement learning models to the field of automatic tuning of quantum compilation, a distributed reinforcement learning (DRL) driven automatic tuning method for quantum compilation is proposed. By decoupling experience generation from agent training, parallel experience generation is achieved based on a distributed cluster. In this method, a markov decision process (MDP) model of quantum compilation is established with the characteristic of dense rewards, a decoupling mechanism for experience generation and agent trainingis is designed, and a dynamic experience loading strategy is combined to improve the training efficiency while ensuring the optimization effect. Experiments demonstrate a 54.6% reduction in training time compared to baseline methods. In terms of optimization performance, the agent performs better than the Qiskit-O3 compiler on 77.3% of the quantum circuits in the test set, and the number of quantum gates of the unseen Shor algorithm circuits is reduced by an average of 17.4%.
RL是机器学习领域的重要范式,其核心思想是通过智能体与环境的持续交互来学习最优策略。作为人工智能领域的研究热点,RL在多个领域(如机器人控制、计算机网络等)展现了强大的潜力[10-12]。智能体在时刻t观测环境状态st ∈S,根据当前策略选择动作at ∈A,执行动作后获得即时奖励rt ∈R并转移到新状态s' t +1。该过程可形式化为MDP四元组(S, A, S', R),其中:S为状态空间;A为动作空间;S'为状态转换后的状态;R:S×A→R为奖励函数。
JAVADI-ABHARIA, TREINISHM, KRSULICHK, et al. Quantum computing with Qiskit[DB/OL]. (2024-06-19)[2025-03-09].
[2]
SIVARAJAHS, DILKESS, COWTANA, et al. T|ket: a retargetable compiler for NISQ devices[J]. Quantum Science and Technology, 2021,6(1):No.014003.
[3]
WIERINGM A, VAN OTTERLOM. Reinforce-ment learning[J]. Adaptation, Learning, and Optimization, 2012,12(3):No.729.
[4]
HAJ-ALIA, HUANGQ J, XIANGJ, et al. Auto-phase: juggling HLS phase orderings in random forests with deep reinforcement learning[J]. Proceedings of machine learning and systems, 2020,2:70-81.
[5]
PUTERMANM L. Markov decision processes[J]. Handbooks in Operations Research and Management Science, 1990,2:331-434.
[6]
QUETSCHLICHN, BURGHOLZERL, WILLER. MQT predictor: automatic device selection with device-specific circuit compilation for quantum computing[J]. ACM Transactions on Quantum Computing, 2025,6(1):1-26.
[7]
SCHULMANJ, WOLSKIF, DHARIWALP, et al. Proximal policy optimization algorithmst[DB/OL]. (2017-08-28)[2025-03-09].
[8]
QUETSCHLICHN, BURGHOLZERL, WILLER. Reducing the compilation time of quantum circuits using pre-compilation on the gate level[C]//Proceedings of the 2023 IEEE International Conference on Quantum Computing and Engineering. Piscataway, USA: IEEE, 2023:757-767.
[9]
MNIHV, KAVUKCUOGLUK, SILVERD, et al. Playing atari with deep reinforcement learning[DB/OL]. (2013-12-19)[2025-03-09].
WATKINSC J C H, DAYANP. Q-learning[J]. Machine Learning, 1992,8(3):279-292.
[14]
VAN HASSELTH, GUEZA, SILVERD. Deep reinforcement learning with double Q-learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016.DOI:10.1609/aaai.v30i1.10295
[15]
WANGZ, SCHAULT, HESSELM, et al. Dueling network architectures for deep reinforcement learning[C]//Proceedings of The International Conference on Machine Learning. New York, USA: PMLR, 2016: 1995-2003.