双空间令牌化与对比增强的异质图Transformer

Contrastive-enhanced Heterogeneous Graph Transformer with Dual-space Tokenization

摘要

近年来,异质图神经网络在处理多类型节点和边的复杂关系方面展现出强大能力,但基于消息传递的架构仍面临表达能力受限、过平滑和过挤压等问题.本文提出基于Transformer架构的CHGormer模型,通过融合对比学习与异构关系编码的令牌生成机制,创新性地解决了异质图中局部异构关系与全局语义依赖的整合难题.具体而言,CHGormer设计了双空间令牌生成策略,在属性与拓扑特征空间中分别采样正负令牌序列,并通过类型感知的邻域聚合生成异构关系表征,将其作为注意力偏置引入全局交互.此外,基于对比学习的跨序列优化进一步增强了节点表示的判别性.在DBLP、Freebase和AMiner 3个基准数据集上的实验表明,CHGormer在节点分类任务中表现优异.本研究为异质图表示学习提供了新思路,并在社交推荐和知识推理等场景中展现出应用潜力.

Abstract

In recent years,Heterogeneous Graph Neural Networks have demonstrated strong capabilities in modeling complex relationships involving multiple types of nodes and edges.However,message-passing-based architectures still face limitations such as restricted expressive power,over-smoothing,and over-squashing.This paper proposes a novel Transformer architecture,CHGormer,which innovatively addresses the challenge of integrating local heterogeneous relations with global semantic dependencies in heterogeneous graphs by combining contrastive learning with a token generation mechanism for heterogeneous relation encoding.Specifically,CHGormer introduces a dual-space token generation strategy,where positive and negative token sequences are sampled separately from the attribute and topology feature spaces.These are then aggregated through type-aware neighborhood aggregation to form heterogeneous relational representations,which are incorporated into global interactions as attention biases.Moreover,a contrastive learning-based cross-sequence optimization is employed to further enhance the discriminative power of node representations.

关键词

图神经网络 / Transformer / 对比学习 / 异质图 / 节点分类

Key words

graph neural networks / Transformer / contrastive learning / heterogeneous graph / node classification

[1] Sun Q,Peng H,Li J,et al.Pairwise learning for name disambiguation in large-scale heterogeneous academic networks[C]//International Conference on Data Mining,2020:511-520.
[2] Kong X,Shi Y,Yu S,et al.Academic social networks:modeling,analysis,mining and applications[J].Journal of Network and Computer Applications,2019,132:86-103,doi:10.1016/j.jnca.2019.01.029.
[3] Pan L,Dai X,Huang S,et al.Academic paper recommendation based on heterogeneous graph[C]//China National Conference on Chinese Computational Linguistics,2025:381-392.
[4] Tang J,Zhang J,Yao L,et al.Arnetminer:extraction and mining of academic social networks[C]//14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,2008:990-998.
[5] Van den Brink M,Benschop Y.Gender in academic networking:the role of gatekeepers in professorial recruitment[J].Journal of Management Studies,2014,51(3):460-492.
[6] Salamat A,Luo X,Jafari A.HeteroGraphRec:a heterogeneous graph-based neural networks for social recommendations[J].Knowledge-Based Systems,2021,217:106817,doi:10.1016/j.knosys.2021.106817.
[7] Wang Y,Sun H,Zhao Y,et al.A heterogeneous graph embedding framework for location-based social network analysis in smart cities[J].IEEE Transactions on Industrial Informatics,2019,16(4):2747-2755.
[8] Cai D,Shao Z,He X,et al.Mining hidden community in heterogeneous social networks[C]//3rd International Workshop on Link Discovery,2005:58-65.
[9] Milroy L,Llamas C.Social networks[J].The Handbook of Language Variation and Change,2013:407-427,doi:10.1002/9780470756591.
[10] Li Z,Liu H,Zhang Z,et al.Learning knowledge graph embedding with heterogeneous relation attention networks[J].IEEE Transactions on Neural Networks and Learning Systems,2021,33(8):3961-3973.
[11] Asprino L,Daga E,Gangemi A,et al.Knowledge graph construction with a façade:a unified method to access heterogeneous data sources on the web[J].ACM Transactions on Internet Technology,2023,23(1):1-31.
[12] Chen X,Jia S,Xiang Y.A review:knowledge reasoning over knowledge graph[J].Expert Systems with Applications,2020,141:112948,doi:10.1016/j.eswa.2019.112948.
[13] Hogan A,Blomqvist E,Cochez M,et al.Knowledge graphs[J].ACM Computing Surveys,2021,54(4):1-37.
[14] Fensel D,imiek U,Angele K,et al.Introduction:what is a knowledge graph?[J].Knowledge Graphs:Methodology,Tools and Selected Use Cases,2020:1-10,doi:10.1007/978-3-030-37439-6_1.
[15] Chen Z,Wang Y,Zhao B,et al.Knowledge graph completion:a review[J].IEEE Access,2020,8:192435-192456,doi:10.1109/ACCESS.2020.3030076.
[16] Yu L,Sun L,Du B,et al.Heterogeneous graph representation learning with relation awareness[J].IEEE Transactions on Knowledge and Data Engineering,2022,35(6):5935-5947.
[17] Chan T H,Cendra F J,Ma L,et al.Histopathology whole slide image analysis with heterogeneous graph representation learning[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition,2023:15661-15670.
[18] Shao Z,Xu Y,Wei W,et al.Heterogeneous graph neural network with multi-view representation learning[J].IEEE Transactions on Knowledge and Data Engineering,2022,35(11):11476-11488.
[19] Yang X,Yan M,Pan S,et al.Simple and efficient heterogeneous graph neural network[C]//AAAI Conference on Artificial Intelligence,2023:10816-10824.
[20] Zhao J,Wang X,Shi C,et al.Heterogeneous graph structure learning for graph neural networks[C]//AAAI Conference on Artificial Intelligence,2021:4697-4705.
[21] Chen M,Huang C,Xia L,et al.Heterogeneous graph contrastive learning for recommendation[C]//16th ACM International Conference on Web Search and Data Mining,2023:544-552.
[22] Wang X,Bo D,Shi C,et al.A survey on heterogeneous graph embedding:methods,techniques,applications and sources[J].IEEE Transactions on Big Data,2022,9(2):415-436.
[23] Yang C,Xiao Y,Zhang Y,et al.Heterogeneous network representation learning:a unified framework with survey and benchmark[J].IEEE Transactions on Knowledge and Data Engineering,2020,34(10):4854-4873.
[24] Zhang K,Wang W,Zhang H,et al.Learning to represent programs with heterogeneous graphs[C]//30th IEEE/ACM International Conference on Program Comprehension,2022:378-389.
[25] Dong Y,Hu Z,Wang K,et al.Heterogeneous network representation learning[C]//29th International Joint Conference on Artificial Intelligence,2020:4861-4867.
[26] Chang Y,Chen C,Hu W,et al.Megnn:meta-path extracted graph neural network for heterogeneous graph representation learning[J].Knowledge-Based Systems,2022,235:107611,doi:10.1016/j.knosys.2021.107611.
[27] Zhang L,Guo J,Bai Q,et al.Dynamic heterogeneous graph representation learning with neighborhood type modeling[J].Neurocomputing,2023,533:46-60,doi:10.1016/j.neucom.2023.02.060.
[28] Zhao B W,Hu L,You Z H,et al.HINGRL:predicting drug-disease associations with graph representation learning on heterogeneous information networks[J].Briefings in Bioinformatics,2022,23(1):bbab515,doi:10.1093/bib/bbab515.
[29] Zhang C,Song D,Huang C,et al.Heterogeneous graph neural network[C]//25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,2019:793-803.
[30] Schlichtkrull M,Kipf T N,Bloem P,et al.Modeling relational data with graph convolutional networks[C]//The Semantic Web:15th International Conference ESWC,2018:593-607.
[31] Liu Y,Zheng Y,Zhang D,et al.Beyond smoothing:unsupervised graph representation learning with edge heterophily discriminating[C]//AAAI Conference on Artificial Intelligence,2023:4516-4524.
[32] Keriven N.Not too little,not too much:a theoretical analysis of graph(over)smoothing[J].Advances in Neural Information Processing Systems,2022,35:2268-2281,doi:10.5555/3600270.3600435.
[33] Lu Z,Fang Y,Yang C,et al.Heterogeneous graph transformer with poly-tokenization[C]//33rd International Joint Conferences on Artificial Intelligence,2024:2234-2242.
[34] Mao Q,Liu Z,Liu C,et al.Hinormer:representation learning on heterogeneous information networks with graph transformer[C]//ACM Web Conference,2023:599-610.
[35] Tang J,Yang Y,Wei W,et al.Higpt:Heterogeneous graph language model[C]//30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,2024:2842-2853.
[36] Chen J,Liu H,Hopcroft J,et al.Leveraging contrastive learning for enhanced node representations in tokenized graph transformers[J].Advances in Neural Information Processing Systems,2024,37:85824-85845,doi:10.52202/079017-2725.
[37] Zhang Y,Gao S,Pei J,et al.Improving social network embedding via new second-order continuous graph neural networks[C]//28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,2022:2515-2523.
[38] Kumar S,Mallik A,Khetarpal A,et al.Influence maximization in social networks using graph embedding and graph neural network[J].Information Sciences,2022,607:1617-1636,doi:10.1016/j.ins.2022.06.075.
[39] Jain L,Katarya R,Sachdeva S.Opinion leaders for information diffusion using graph neural network in online social networks[J].ACM Transactions on the Web,2023,17(2):1-37.
[40] Yang L,Wang S,Tao Y,et al.Dgrec:graph neural network for recommendation with diversified embedding generation[C]//16th ACM International Conference on Web Search and Data Mining,2023:661-669.
[41] Wu Z,Zhou J,Zhang J,et al.A deep prediction framework for multi-source information via heterogeneous GNN[C]//30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,2024:3460-3471.
[42] Lu Y,Gao M,Liu H,et al.Neighborhood overlap-aware heterogeneous hypergraph neural network for link prediction[J].Pattern Recognition,2023,144:109818,doi:10.1016/j.patcog.2023.109818.
[43] Rampásˇek L,Galkin M,Dwivedi V P,et al.Recipe for a general,powerful,scalable graph transformer[C]//36th International Conference on Neural Information Processing Systems,2022:14501-14515.
[44] Yun S,Jeong M,Kim R,et al.Graph transformer networks[C]//33rd International Conference on Neural Information Processing Systems,2019:11983-11993.
[45] Ying C,Cai T,Luo S,et al.Do transformers really perform badly for graph representation?[J].Advances in Neural Information Processing Systems,2021,34:28877-28888,doi:10.5555/3540261.354273.
[46] Hu Z,Dong Y,Wang K,et al.Heterogeneous graph transformer[C]//Proceedings of the Web Conference,2020:2704-2710.
[47] Kipf T N,Welling M.Semi-supervised classification with graph convolutional networks[C]//International Conference on Learning Representations,2017:1-14,doi:10.48550/arXiv.1609.02907.
[48] Wang X,Ji H,Shi C,et al.Heterogeneous graph attention network[C]//World Wide Web Conference,2019:2022-2032.
[49] Fu X,Zhang J,Meng Z,et al.Magnn:metapath aggregated graph neural network for heterogeneous graph embedding[C]//Proceedings of the Web Conference,2020:2331-2341.
[50] Zhang M,Wang X,Zhu M,et al.Robust heterogeneous graph neural networks against adversarial attacks[C]//AAAI Conference on Artificial Intelligence,2022:4363-4370.
[51] Fu X,King I.MECCH:metapath context convolution-based heterogeneous graph neural networks[J].Neural Networks,2024,170:266-275,doi:10.1016/j.neunet.2023.11.030.
[52] Tian Y,Dong K,Zhang C,et al.Heterogeneous graph masked autoencoders[C]//AAAI Conference on Artificial Intelligence,2023:9997-10005.
[53] Zhu S,Zhou C,Pan S,et al.Relation structure-aware heterogeneous graph neural network[C]//International Conference on Data Mining,2019:1534-1539.
[54] Lv Q,Ding M,Liu Q,et al.Are we really making much progress?revisiting,benchmarking and refining heterogeneous graph neural networks[C]//27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,2021:1150-1160.
[55] Özcan Şimşek N Ö,Özgür A,Gürgen F.GNNMutation:a heterogeneous graph-based framework for cancer detection[J].BMC Bioinformatics,2025,26(1):1-17.
[56] Corso G,Cavalleri L,Beaini D,et al.Principal neighbourhood aggregation for graph nets[J].Advances in Neural Information Processing Systems,2020,33:13260-13271,doi:10.5555/3495724.3496836.
[57] Jiao P,Yu K,Bao Q,et al.Graph contrastive learning with node-level accurate difference[J].Fundamental Research,2025,5(2):818-829.
[58] Guo X,Jiao P,Shi D,et al.Learning node representations via sketching the generative process with events benefits link prediction on heterogeneous networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2025,47(5):3961-3974.
[59] Wang X,Liu N,Han H,et al.Self-supervised heterogeneous graph neural network with co-contrastive learning[C]//27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining,2021:1726-1736.
[60] Lin T,Zhou C,Li Q,et al.Multi-level disentangled contrastive learning on heterogeneous graphs[C]//16th International Conference on Machine Learning and Computing,2024:628-634.
[61] Gui H,Liu J,Tao F,et al.Embedding learning with events in heterogeneous information networks[J].IEEE Transactions on Knowledge and Data Engineering,2017,29(11):2428-2441.

基金资助

国家自然科学基金项目(62372146)资助;浙江省自然科学基金项目(LDT23F01015F01)资助.