双空间令牌化与对比增强的异质图 Transformer

Contrastive-enhanced Heterogeneous Graph Transformer with Dual-space Tokenization

摘要

近年来，异质图神经网络在处理多类型节点和边的复杂关系方面展现出强大能力，但基于消息传递的架构仍面临表达能力受限、过平滑和过挤压等问题。本文提出基于 Transformer 架构的 CHGormer 模型，通过融合对比学习与异构关系编码的令牌生成机制，创新性地解决了异质图中局部异构关系与全局语义依赖的整合难题。具体而言，CHGormer 设计了双空间令牌生成策略，在属性与拓扑特征空间中分别采祥正负令牌序列，并通过类型感知的邻域聚合生成异构关系表征，将其作为注意力偏置引入全局交王。此外，基于对比学习的跨序列优化进一步增强了节点表示的判别性。在 DBLP、Freebase 和 AMiner 3 个基准数据集上的实验表明，CHGormer 在节点分类任务中表现优异。本研究为异质图表示学习提供了新思路，并在社交推荐和知识推理等场景中展现出应用潜力。

Abstract

In recent years，Heterogeneous Graph Neural Networks have demonstrated strong capabilities in modeling complex relation- ships involving multiple types of nodes and edges.However，message-passing-based architectures still face limitations such as restricted expressive power，over-smoothing，and over-squashing.This paper proposes a novel Transformer architecture，CHGormer，which inno- vatively addresses the challenge of integrating local heterogeneous relations with global semantic dependencies in heterogeneous graphs by combining contrastive learning with a token generation mechanism for heterogeneous relation encoding.Specifically，CHGormer in- troduces a dual-space token generation strategy，where positive and negative token sequences are sampled separately from the attribute and topology feature spaces.These are then aggregated through type-aware neighborhood aggregation to form heterogeneous relational representations，which are incorporated into global interactions as attention biases.Moreover，a contrastive learning-based cross-se- quence optimization is employed to further enhance the discriminative power of node representations.

关键词

图神经网络 / Transformer / 对比学习 / 异质图 / 节点分类

Key words

graph neural networks / Transformer / contrastive learning / hetcrogencous graph / node classification

参考文献

[1] Sun Q, Peng H, Li J, et al. Pairwise learning for name disambigu- ation in large-scale heterogeneous academic networks[C]// Interna- tional Conference on Data Mining, 2020:511-520.

[2] Kong X, Shi Y, Yu S, et al. Academic social networks:modeling,a- nalysis,mining and applications[J]. Journal of Network and Com- puter Applications, 2019, 132:86-103,doi:10.1016/j.jnca.2019.01.029.

[3] Pan L, Dai X, Huang S, et al. Academic paper recommendation based on heterogeneous graph[C]// China National Conference on Chinese Computational Linguistics, 2025:381-392.

[4] Tang J, Zhang J, Yao L, et al. Arnetminer:extraction and mining of academic social networks[C]// 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2008:990-998.

[5] Van den Brink M, Benschop Y. Gender in academic networking:the role of gatekeepers in professorial recruitment[J]. Journal of Man- agement Studies, 2014, 51(3):460-492.

[6] Salamat A, Luo X, Jafari A. HeteroGraphRec:a heterogeneous graph-based neural networks for social recommendations[J]. Knowledge-Based Systems, 2021,217:106817,doi:10.1016/j.kn-osys.2021.106817.

[7] Wang Y, Sun H, Zhao Y, et al. A heterogeneous graph embedding framework for location-based social network analysis in smart cities[J]. IEEE Transactions on Industrial Informatics, 2019, 16 (4): 2747-2755.

[8] Cai D, Shao Z, He X, et al. Mining hidden community in heteroge- neous social networks[C]// 3rd International Workshop on Link Discovery, 2005:58-65.

[9] Milroy L, Llamas C. Social networks[J]. The Handbook of Lan- guage Variation and Change, 2013:407-427,doi:10.1002/9780470756591.

[10] Li Z, Liu H, Zhang Z, et al. Leaming knowledge graph embedding with heterogeneous relation attention networks[J]. IEEE Transac- tions on Neural Networks and Learning Systems, 2021, 33 (8): 3961-3973.

[11] Asprino L, Daga E, Gangemi A, et al. Knowledge graph construc- tion with a facade:a unified method to access heterogeneous data sources on the web[J]. ACM Transactions on Internet Technolo- gy, 2023, 23(1):1-31.

[12] Chen X, Jia S, Xiang Y. A revicw:knowledge reasoning over knowledge graph[J]. Expert Systems with Applications, 2020, 141:112948,doi:10.1016/j.eswa.2019.112948.

[13] Hogan A, Blomqvist E, Cochez M, et al. Knowledge graphs[J]. ACM Computing Surveys, 2021, 54(4):137.

[14] Fensel D, Şimşiek U, Angele K, et al. Introduction:what is a knowl- edge graph?[J]. Knowledge Graphs:Methodology,Tools and Se- lected Use Cases, 2020:1-10,doi:10.1007/978-3-030-37439-6_1.

[15] Chen Z, Wang Y, Zhao B, et al. Knowledge graph completion:a re- view[J]. IEEE Access, 2020, 8 :192435-192456,doi:10.1109/ACCESS.2020.3030076.

[16] Yu L, Sun L, Du B, et al. Heterogeneous graph representation learn- ing with relation awareness[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 35(6):5935-5947.

[17] Chan T H, Cendra F J, Ma L, et al. Histopathology whole slide im- age analysis with heterogeneous graph representation learning[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023:15661-15670.

[18] Shao Z, Xu Y, Wei W, et al. Heterogeneous graph neural network with multi-view representation learning[J]. IEEE Transactions on Knowlcdgc and Data Enginccring, 2022, 35(11):11476-11488.

[19] Yang X, Yan M, Pan S, et al. Simple and efficient heterogeneous graph neural network[C]// AAAI Conference on Artificial Intelli- gence, 2023:10816-10824.

[20] Zhao J, Wang X, Shi C, et al. Heterogeneous graph structure learn ing for graph neural networks[C]// AAAI Conference on Artificial Intelligence, 2021:4697-4705.

[21] Chen M, Huang C, Xia L, et al. Heterogeneous graph contrastive learning for recommendation[C]// 16th ACM International Con- ference on Web Search and Data Mining, 2023:544-552.

[22] Wang X, Bo D, Shi C, et al. A survey on heterogeneous graph em- bedding:methods,techniques,applications and sources[J]. IEEE Transactions on Big Data, 2022, 9(2):415-436.

[23] Yang C, Xiao Y, Zhang Y, et al. Heterogeneous network representa- tion learning:a unified framework with survey and benchmark[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 34 (10):4854-4873.

[24] Zhang K, Wang W, Zhang H, et al. Learning to represent programs with heterogeneous graphs[C]// 30th IEEE/ACM International Conference on Program Comprehension, 2022:378-389.

[25] Dong Y, Hu Z, Wang K, et al. Heterogeneous network representa- tion learning[C]// 29th International Joint Conference on Artificial Intelligence, 2020:4861-4867.

[26] Chang Y, Chen C, Hu W, et al. Megnn:meta-path extracted graph neural network for heterogeneous graph representation learning[J]. Knowledge-Based Systems, 2022,235:107611,doi:10.1016/j.knosys.2021.107611.

[27] Zhang L, Guo J, Bai Q, et al. Dynamic heterogeneous graph repre- sentation learning with neighborhood type modeling[J]. Neurocom- puting, 2023, 533:46-60,doi:10.1016/j.neucom.2023.02.060.

[28] Zhao B W, Hu L, You Z H, et al. HINGRL:predicting drug-disease associations with graph representation learning on heterogeneous in- formation networks[J]. Briefings in Bioinformatics, 2022, 23(1): bbab515,doi:10.1093/bib/bbab515.

[29] Zhang C, Song D, Huang C, et al. Heterogeneous graph neural net- work[C]// 25th ACM SIGKDD International Conference on Knowledge Discovery ＆ Data Mining, 2019:793-803.

[30] Schlichtkrull M, Kipf T N, Bloem P, et al. Modeling relational data with graph convolutional networks[C]// The Semantic Web: 15th International Conference ESWC, 2018:593-607.

[31] Liu Y, Zheng Y, Zhang D, et al. Beyond smoothing: unsupervised graph representation learning with edge heterophily discriminating[C]// AAAI Conference on Artificial Intelligence, 2023:4516-4524.

[32] Keriven N. Not too little,not too much:a theoretical analysis of graph(over)smoothing[J]. Advances in Neural Information Pro- cessing Systems, 2022, 35:2268-2281,doi:10.5555/3600270.3600435.

[33] Lu Z, Fang Y, Yang C ct al. Hetcrogencous graph transformcr with poly-tokenization[C]// 33rd International Joint Conferences on Artificial Intelligence, 2024:2234-2242.

[34] Mao Q, Liu Z, Liu C, et al. Hinormer:representation learning on heterogeneous information networks with graph transformer[C]// ACM Web Conference, 2023:599-610.

[35] Tang J, Yang Y, Wei W, et al. Higpt:Heterogeneous graph language model[C]// 30th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, 2024:2842-2853.

[36] Chen J, Liu H, Hopcroft J, et al. Leveraging contrastive learning for enhanced node representations in tokenized graph transformers[J]. Advances in Neural Information Processing Systems, 2024, 37: 85824-85845,doi: 10.52202/079017-2725.

[37] Zhang Y, Gao S, Pei J, et al. Improving social network embedding via new second-order continuous graph neural networks[C]// 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,2022:2515-2523.

[38] Kumar S, Mallik A, Khetarpal A, et al. Influence maximization in social networks using graph embedding and graph neural network[J]. Information Sciences, 2022, 607:1617-1636,doi:10.1016/j.ins.2022.06.075.

[39] Jain L, Katarya R, Sachdeva S. Opinion leaders for information dif- fusion using graph neural network in online social networks[J]. ACM Transactions on the Web, 2023, 17(2):1-37.

[40] Yang L, Wang S, Tao Y, et al. Dgrec :graph neural network for rec- ommendation with diversified embedding generation[C]// 16th ACM International Conference on Web Search and Data Mining, 2023:661-669.

[41] Wu 7 Thou I, Zhang I, et al. A deep prediction framework for multi-source information via heterogeneous GNN[C]// 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024:3460-3471.

[42] Lu Y, Gao M, Liu H, et al. Neighborhood overlap-aware heterogene- ous hypergraph neural network for link prediction[J]. Pattern Rec- ognition, 2023,144:109818,doi:10.1016/j.patcog.2023.109818.

[43] Rampášek L, Galkin M, Dwivedi V P, et al. Recipe for a general,pow- erful,scalable graph transformer[C]// 36 th International Conference on Neural Information Processing Systems,2022:14501-14515.

[44] Yun S, Jeong M, Kim R, et al. Graph transformer networks[C]// 33rd International Conference on Neural Information Processing Systems, 2019:11983-11993.

[45] Ying C, Cai T, Luo S, et al. Do transformers really perform badly for graph representation?[J]. Advances in Neural Information Processing Systems, 2021,34: 28877 28888,doi:10.5555/3540261.354273.

[46] Hu Z, Dong Y, Wang K, et al. Heterogeneous graph transformer[C]// Proceedings of the Web Conference, 2020:2704-2710.

[47] Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[C]// International Conference on Learning Representations, 2017:1-14,doi:10.48550/arXiv.1609.02907.

[48] Wang X, Ji H, Shi C, et al. Heterogeneous graph attention network[C]// World Wide Web Conference, 2019:2022-2032.

[49] Fu X, Zhang J, Meng Z, et al. Magnn:metapath aggregated graph neural network for heterogeneous graph embedding[C]// Proceed- ings of the Web Conference, 2020:2331-2341.

[50] Zhang M, Wang X, Zhu M, et al. Robust heterogeneous graph neu- ral networks against adversarial attacks[C]// AAAI Conference on Artificial Intelligence, 2022:4363-4370.

[51] Fu X, King I. MECCH: metapath context convolution-based hetero-geneous graph neural networks[J]. Neural Networks, 2024, 170: 266-275,doi:10.1016/j.neunet.2023.11.030.

[52] Tian Y, Dong K, Zhang C, et al. Heterogeneous graph masked au- toencoders[C]// AAAI Conference on Artificial Intelligence, 2023:9997-10005.

[53] 7hu S, Thou C, Pan S, et al. Relation structure-aware heterogeneous graph neural network[C]// International Conference on Data Min- ing, 2019:1534-1539.

[54] Lv Q, Ding M, Liu Q, et al. Are we really making much progress? revisiting,benchmarking and refining heterogeneous graph neural networks[C]// 27th ACM SIGKDD Conference on Knowledge Discovery ＆ Data Mining, 2021:1150-1160.

[55] Özcan Şimşek N Ö, Özgür A, Gürgen F. GNNMutation:a heteroge- neous graph based framework for cancer detection[J]. BMC Bioin formatics, 2025, 26(1):1-17.

[56] Corso G, Cavalleri L, Beaini D, et al. Principal neighbourhood aggre- gation for graph nets[J]. Advances in Neural Information Processing Systems, 2020, 33:13260-13271,doi: 10.5555/3495724.3496836.

[57] Jiao P, Yu K, Bao Q, et al. Graph contrastive learning with node- level accurate difference[J]. Fundamental Research, 2025, 5(2): 818-829.

[58] Guo X, Jiao P, Shi D, et al. Learning node representations via sketc- hing the generative process with events benefits link prediction on heterogeneous networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(5):3961-3974.

[59] Wang X, Liu N, Han H, et al. Self-supervised heterogeneous graph neural network with co-contrastive learning[C]// 27 th ACM SIGKDD Conference on Knowledge Discovery ＆ Data Mining, 2021:1726-1736.

[60] Lin T, Zhou C, Li Q, et al. Multi-level disentangled contrastive learning on heterogeneous graphs[C]// 16th International Confer- ence on Machine Learning and Computing,2024:628-634.

[61] Gui II, Liu J, Tao F, et al. Embedding learning with events in heter- ogeneous information networks[J]. IEEE Transactions on Knowl- edge and Data Engineering, 2017, 29(11):2428-2441.

基金资助

国家白然科学基金项目(62372146)

浙江省白然科学基金项目(LDT23F01015F01)