基于 Transformer 和强化学习的时序知识图谱补全方法

Temporal Knowledge Graph Completion Method Based on Transformer and Reinforcement Learning

摘要

针对现有时序知识图谱推理存在捕获长距离依赖关系能力不足以及可解释性缺乏的问题，提出了一种结合 Transform- er 和强化学习的时序知识图谱补全模型。该模型使用强化学习设计了一种具有高度解释性的新型策略网络，由时序感知编码器、路径上下文编码器和动作评分器 3 个核心组件构成。首先，时序感知编码器利用自注意力机制将时间信息嵌入到关系表示中，增强了对时间动态性的处理能力；其次，路径上下文编码器利用 Transformer 高效编码历史事件序列，捕捉了长距离依赖关系；再次，动作评分器利用双向门控循环单元进行动作预测，提升了预测准确性。此外，针对奖励稀疏性问题，所提模型引入了一种新型奖励函数，综合考虑时间塑形奖励、路径长度奖励以及路径多祥性奖励，提供更细致的反馈以优化路径选择.本文在 4 个公开数据集上与现有先进方法进行比较，结果表明所提模型在评估指标 MRR 和 Hits@k 上较基线方法均有所提升。

Abstract

Aiming at the problem that the existing temporal knowledge graph reasoning has insufficient ability to capture long-distance dependencies and lack of interpretability，a temporal knowledge graph completion model combining Transformer and reinforcement learning is proposed.This model uses reinforcement learning to design a new highly interpretive strategy network，which is composed of three core components；time-aware encoder，path context encoder and action scoring device.First，the time-aware encoder uses the self attention mechanism to embed the time information into the relational representation，which enhances the ability to deal with the time dynamics；Secondly，the path context encoder uses Transformer to efficiently encode historical event sequences，capturing long- distance dependencies；Thirdly，the action scorer uses the two-way gated cycle unit to predict actions，which improves the accuracy of prediction.In addition，for the problem of reward sparsity，the proposed nodel introduces a new reward function，which comprehensive- ly considers time shaping reward，path length reward and path diversity reward，and provides more detailed feedback to optimize path selection.This paper compares the proposed model with the existing advanced methods on four public datasets，and the results show that the proposed model is effective in evaluating the MRR and [e-mail] with the baseline method，the above method has improved.

关键词

时序知识图谱补全 / 强化学习 / Transformer / 奖励函数

Key words

temporal knowledge graph completion / reinforcement learning / Transformer / reward function

参考文献

[1] MA H K, QI Y S, WU Y B. Knowledge graph completion method based on disentangled neighborhood information aggregation[J]. Applied Research of Computers, 2024, 41(3):772-778.

[2] LI C Y, WU Y Q, TANG Z K, et al. Learner personalized resource recommendation based on knowledge graph[J]. Journal of Chinese Computer Systems, 2024, 45(2):285-292.

[3] XIAO L, LI Q. Survey on time series knowledge graph completion methods[J]. Computer Engineering and Application, 2024, 60(6): 43-54.

[4] SHEN Y H, JIANG X H, WANG Y Z, et al. Review of reasoning on temporal knowledge graphs[J]. Chinese Journal of Computer, 2023, 46(6):1272-1301.

[5] WANG Y H, CHEN Z Y, 7HAO X, et al. Research progress and trend of temporal knowledge graph representation and reasoning[J]. Journal of Software, 2024, 35(8):3923-3951.

[6] Leblay J, Chekol M W. Deriving validity time in knowledge graph[C]//Companion Proceedings of the Web Conference, 2018: 1771-1776.

[7] Ma Y, Tresp V, Daxberger E A. Embedding models for episodic knowledge graphs[J]. Journal of Web Semantics, 2019, 59: 100490,doi:10.1016/j.websem.2018.12.008

[8] Zhu C, Chen M, Fan C, et al. Learning from history:modeling tem- poral knowledge graphs with sequential copy-generation networks[C]//Proceedings of the AAAI Conference on Artificial Intelli- gence, 2021:4732-4740.

[9] Wan G, Pan S, Gong C, et al. Reasoning like human: hierarchical reinforcement learning for knowledge graph reasoning[C]//Inter-national Joint Conference on Artificial Intelligence, International Joint Conference on Artificial Intelligence,2021:1926-1932.

[10] Zheng S, Chen W, Zhao P, et al. When hardness makes a differ- ence:multi-hop knowledge graph reasoning over few-shot relations[C]//Proceedings of the 30th ACM International Conference on Information ＆ Knowledge Management, 2021:2688-2697

[11] Tao Y, Li Y, Wu Z. Temporal link prediction via reinforcement learning[C]//IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP),2021:3470-3474.

[12] Bai L, Yu W, Chen M, et al. Multi-hop reasoning over paths in temporal knowledge graphs using reinforcement learning[J]. Ap- plied Soft Computing, 2021,103:107144,doi:10.1016/j.asoc.2021.107144.

[13] Sun H, Zhong J, Ma Y, et al. Timetraveler:reinforcement learning for temporal knowledge graph forecasting[C]//Empirical Methods in Natural Language Processing, 2021:8306-8319.

[14] LI Z L, CIIEN Q, SIII L, et al. Dynamic knowledge graph comple- tion based on time-aware combination[J]. Journal of Zhejiang Uni- versity:Engineering Science, 2024, 58(8):1738-1747.

[15] Wu J, Cao M, Cheung J C K, et al. Temp:temporal message pass- ing for temporal knowledge graph completion[C]//Empirical Methods in Natural Language Processing, 2020:5730-5746.

[16] Jin W, Qu M, Jin X, et al. Recurrent event network:autoregressive structure inference over temporal knowledge graphs[ C]//Empiri- cal Methods in Natural Language Processing, 2020:6669-6683.

[17] Sun H, Geng S, Zhong J, et al. Graph hawkes transformer for ex- trapolated reasoning on temporal knowledge graphs[C]//Proceed- ings of the Conference on Empirical Methods in Natural Language Processing, 2022:7481-7493.

[18] Park N, Liu F, Mehta P, et al. Evokg:jointly modeling event time and network structure for reasoning over temporal knowledge graphs[C]//Proceedings of the 15th ACM International Confer- ence on Web Search and Data Mining, 2022:794-803.

[19] Li Y, Sun S, Zhao J. TiRGN:time-guided recurrent graph network with local-global historical patterns for temporal knowledge graph reasoning[C]//International Joint Conference on Artificial Intelli- gence( IJCAI), 2022:2152-2158.

[20] Li C, Qiu M. Reinforcement learning for cyber-physical systems: with cybersecurity case studies[M]. Chapman and Hall/CRC, 2019.

[21] Yi X, Junyong L, Mingjing L, et al. Reason more like human:in- corporating meta information into hierarchical reinforcement learn- ing for knowledge graph reasoning[J]. Applied Intelligence, 2022, 53 (11):13293-13308.

[22] Zheng S, Yin H, Chen T, et al. DREAM:adaptive reinforcement learning based on attention mechanism for temporal knowledge graph reasoning[C]//Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023:1578-1588.

[23] Luo X, 7hu A, 7hang I, et al. HierarT:multi-hop temporal knowl- edge graph forecasting with hierarchical reinforcement learning[J]. Knowledge-Based Systems, 2024,300:112164,doi:10.1016/j.knosys.2024.112164.

[24] Liu W, Wang P, Zhang Z, et al. Multi-scale convolutional neural network for temporal knowledge graph completion[J]. Cognitive Computation, 2023, 15(3):1016-1022.

[25] Zhao X, Miao J, Yang F, et al. Geometry interaction embeddings for interpolation temporal knowledge graph completion[J]. Mathe- matics, 2024, 12(13):2022,doi: 10.3390/math12132022.

[26] Gao Y, He Y, Kan Z, et al. Learning joint structural and temporal contextualized knowledge embeddings for temporal knowledge graph completion[C]//Findings of the Association for Computa- tional Linguistics(ACL), 2023:417-430.

[27] Zhang M, Xia Y, Liu Q, et al. Learning latent relations for temporal knowledge graph reasoning[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics( Volume 1: Long Papers), 2023:12617-12631.

[28] Goel R, Kazemi S M, Brubaker M, et al. Diachronic embedding for temporal knowledge graph completion[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020:3988-3995.

[29] Wang J, Wu R F, Wu Y W, et al. MPNet:temporal knowledge graph completion based on a multi-policy network[J]. Applied In- telligence, 2024, 54(3):2491-2507.

[30] 马浩凯, 祁云嵩, 吴宁斌. 解纠缠邻域信息聚合的知识图谱补全方法[J]. 计算机应用研究, 2024, 41(3):772-778.

[31] 李春英, 武毓琦, 汤志康, 等. 融合知识图谱的学习者个性化学习资源推存[J]. 小型微型计算机系统, 2024, 45(2):285-292.

[32] 肖蕓, 李琪. 时序知识图谱补全方法研究综述[J]. 计算机工程与应用, 2024, 60(6):43-54.

[33] 沈英汉, 江旭晖, 干元卓, 等. 时态知识图谱的推理研究综述[J]. 计算机学报, 2023, 46(6):1272-1301.

[34] 王俞涵, 陈子阴, 赵翔, 等. 时序知识图谱表示与推理的研究进展与趋势[J]. 软件学报, 2024, 35(8):3923-3951.

[35] 李忠良, 陈麒, 石琳, 等. 时间感知组合的动态知识图谱补全[J]. 浙江大峃峃报(工峃版), 2024, 58(8):1738-1747.

基金资助

国家自然科学基金青年项目(62201377)

虚拟现实技术与系统国家重点实验室（北京航天航空大学）开放课题基金项目(VRLAB2022C1)

山西省重点研究计划项目(202102020101004)

山西省四国留学人员科研教研项目(2024-61)