融合图文层次多粒度特征的网络谣言检测方法

Method for Network Rumor Detection Integrating Hierarchical Multi-granularity Features of Images and Text

摘要

随着社交媒体的普及,以图文结合为主要形式的多模态谣言已成为网络空间的一大危害.在谣言传播的早期阶段,由于缺乏用户转发、评论等社交上下文信息,对谣言的快速识别面临挑战.当前仅依赖帖子内容的检测方法,往往因对图文内部特征的挖掘不够深入,导致对模态间深层一致性的判断能力有限,从而限制了谣言早期检测的性能.为解决这一问题,本文提出一种层次化多粒度特征融合网络模型HMF-Net.该模型首先通过多分支结构深度提取图文内容的多粒度特征:并行地从文本中获取句子级全局语义与短语级局部特征;同时从图像中提取高层语义、多尺度纹理及频域物理特征.进而,模型利用一种层次化Transformer架构,先完成各模态内部多粒度特征的对齐与预融合,再通过二次聚合网络实现跨模态特征的深度交互与补齐.这种设计能够更充分地捕捉图文间的内在关联与互补信息,从而提升模型在缺少社交数据时的鲁棒性.实验结果表明,本文提出的HMF-Net模型在公开的Weibo和Twitter数据集上表现优异,准确率分别达到91.5%和88.4%,显著优于其他仅依赖内容的基线模型,验证该方法在多模态谣言早期检测任务中的有效性和可行性.

Abstract

With the proliferation of social media,multimodal rumors,primarily in the form of combined text and images,have become a major threat in cyberspace.In the early stages of rumor propagation,the lack of social context information,such as user retweets and comments,poses a significant challenge to rapid identification.Current detection methods that rely solely on post content are often limited in their performance for early detection,as their insufficient mining of internal features restricts their ability to judge deep cross-modal consistency.To address this issue,this paper proposes a hierarchical multi-granularity fusion network,named HMF-Net.The model first deeply extracts multi-granularity features from text-image content via a multi-branch structure:concurrently obtaining sentence-level global semantics and phrase-level local features from text,while simultaneously extracting high-level semantics,multi-scale textures,and frequency-domain physical features from images.Furthermore,the model utilizes a hierarchical Transformer architecture to first perform alignment and pre-fusion of multi-granularity features within each modality,and then achieves deep cross-modal interaction and complementation through a secondary aggregation network.This design allows for a more comprehensive capture of the intrinsic correlations and complementary information between text and image,thereby enhancing the model′s robustness in the absence of social data.Experimental results on the public Weibo and Twitter datasets show that the proposed HMF-Net model exhibits superior performance,achieving accuracies of 91.5% and 88.4% respectively,significantly outperforming other content-only baseline models.This validates the effectiveness and feasibility of the method in the task of multimodal rumor early detection.

关键词

谣言检测 / 多模态融合 / 多粒度特征 / 层次化聚合 / 多层级特征

Key words

rumor detection / multimodal fusion / multi-granularity features / hierarchical aggregation / multi-level feature

[1] Ma J,Gao W,Wong K F.Rumor detection on twitter with tree structured recursive neural networks[C]//56th Annual Meeting of the Association for Computational Linguistics,2018:1980-1989.
[2] Ma J,Gao W,Wei Z Y,et al.Detect rumors using time series of social context information on microblogging websites[C]//24th ACM International Conference on Information and Knowledge Management,2015:1751-1754.
[3] Dou Y T,Liu Z W,Sun L,et al.Enhancing graph neural network-based fraud detectors against camouflaged fraudsters[C]//29th ACM International Conference on Information and Knowledge Management,2020:315-324.
[4] Wang Y Z,Qian S S,Hu J,et al.Fake news detection via knowledge-driven multimodal graph convolutional networks[C]//International Conference on Multimedia Retrieval,2020:540-547.
[5] Hu L M,Yang T C,Zhang L H,et al.Compare to the knowledge:graph neural fake news detection with external knowledge[C]//59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,2021:754-763.
[6] ZHANG X Y,SUN F,FENG L,et al.Multi-view representations for fake news detection[J].Netinfo Security,2024,24(3):438-448.
[7] Wu Y,Zhan P W,Zhang Y J,et al.Multimodal fusion with co-attention networks for fake news detection[C]//Findings of the Association for Computational Linguistics,2021:2560-2569.
[8] Chen Y X,Li D S,Zhang P,et al.Cross-modal ambiguity learning for multimodal fake news detection[C]//ACM Web Conference,2022:2897-2905.
[9] Peng L W,Jian S L,Li D S,et al.MRML:multimodal rumor detection by deep metric learning[C]//IEEE International Conference on Acoustics,Speech and Signal Processing,2023:1-5.
[10] WANG Z Y,ZHU X F.Research on fake news detection based on multimodal transformer[J].Journal of the China Society for Scientific and Technical Information,2023,42(12):1477-1486.
[11] HUANG T,XIAO Y Z,XIANG J P,et al.A multi-domain rumor early detection method fusing multi-level feature representations[J].Journal of Intelligence,2025,44(4):127-135.
[12] Radford A,Kim J W,Hallacy C,et al.Learning transferable visual models from natural language supervision[C]//38th International Conference on Machine Learning,2021:8748-8763.
[13] Lin T Y,Dollár P,Girshick R,et al.Feature pyramid networks for object detection[C]//IEEE Conference on Computer Vision and Pattern Recognition,2017:936-944.
[14] Boididou C,Papadopoulos S,Dang Nguyen D T,et al.Verifying multimedia use at MediaEval 2016[C]//MediaEval Workshop,2016:1-3.
[15] Jin Z W,Cao J,Guo H,et al.Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//25th ACM International Conference on Multimedia,2017:795-816.
[16] Wang Y Q,Ma F L,Jin Z W,et al.EANN:event adversarial neural networks for multi-modal fake news detection[C]//24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,2018:849-857.
[17] Khattar D,Goud J S,Gupta M,et al.MVAE:multimodal variational autoencoder for fake news detection[C]//The World Wide Web Conference(WWW),New York:ACM,2019:2915-2921.
[18] Peng L W,Jian S L,Kan Z G,et al.Not all fake news is semantically similar:contextual semantic representation learning for multimodal fake news detection[J].Information Processing & Management,2024,61(1):103564,doi:10.1016/j.ipm.2023.103564.
[19] Singhal S,Shah R R,Chakraborty T,et al.SpotFake:a multi-modal framework for fake news detection[C]//IEEE 5th International Conference on Multimedia Big Data,2019:39-47.
[20] PEI S L,GUO J S,HAN Q H,et al.A coarse-grained granular ball generation algorithm[J].Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition),2025,37(6):781-791.
[21] WANG A R,YUAN D Y,PAN Y Q,et al.Multi-modal rumor detection model based on dual attention mechanism on hypergraphs[J].Journal of Frontiers of Computer Science and Technology,2025,(11):3037-3045,doi:10.3778/j.issn.1673-9418.2411048.
附中文参考文献:
[6] 张新有,孙 峰,冯 力,等.基于多视图表征的虚假新闻检测[J].信息网络安全,2024,24(3):438-448.
[10] 王震宇,朱学芳.基于多模态Transformer的虚假新闻检测研究[J].情报学报,2023,42(12):1477-1486.
[11] 黄 涛,肖玉芝,向洁萍,等.融合多层级特征表示的多领域谣言早期检测方法[J].情报杂志,2025,44(4):127-135.
[20] 裴生雷,郭建朔,韩清浩,等.一种粗粒度的粒球生成算法[J].重庆邮电大学学报(自然科学版),2025,37(6):781-791.
[21] 王安然,袁得嵛,潘语泉,等.基于超图双重注意力机制的多模态谣言检测模型[J].计算机科学与探索,2025,(11):3037-3045,doi:10.3778/j.issn.1673-9418.2411048.

基金资助

辽宁省自然科学基金项目(2022-MS-168,2019-ZD-0167)资助;中国刑事警察学院重点项目(2024YCZD07)资助;辽宁省教育厅重点科研项目(C2024009)资助;辽宁省社会科学规划基金项目(L24BFX008)资助.