基于自注意力机制的中心距差异多模态情感分析

Center moment discrepancy multimodal sentiment analysis based on self-attention mechanism

摘要

为了解决现有模型在模态间相关性挖掘、特征融合方式和标签更新机制上存在的问题,提出一种基于自注意力机制的中心距差异多模态情感分析方法(center moment discrepancy multimodal sentiment analysis based on self-attention mechanism, SA-CMD)。首先,使用编码器对提取的特征序列进行编码,并通过自注意力机制动态调整各模态特征的权重,捕捉模态间复杂的依赖关系。然后,引入中心距差异方法动态优化特征表示和标签分布,增强模型的鲁棒性。在特征融合过程中,通过计算模态特征与其正负中心的距离差异,生成更准确的特征标签,进一步提高融合特征的质量。最终,使用线性层将融合特征投影到低维空间进行预测。实验结果表明,SA-CMD在公开数据集CMU-MOSI和CMU-MOSEI上的各项评价指标均优于现有基准模型,特别是在相关系数、二分类精度和七分类精度指标上表现优越。进一步验证自注意力机制和中心距差异方法在提升模型性能中的关键作用,充分说明SA-CMD模型在多模态情感分析任务中的有效性和鲁棒性。

Abstract

A center moment discrepancy multimodal sentiment analysis based on self-attention mechanism (SA-CMD) is proposed, aiming to address issues related to modality correlation mining, feature fusion strategies, and label updating mechanisms in existing models. First, an encoder is used to encode the extracted feature sequences, and the weights of each modality’s features are dynamically adjusted through a self-attention mechanism to capture the complex dependencies between modalities. Next, the center moment discrepancy method is introduced to dynamically optimize feature representations and label distributions, enhancing the model’s robustness. During the feature fusion process, the model calculates the distance discrepancy between modality features and their respective positive and negative centers to generate more accurate feature labels, further improving the quality of the fused features. Finally, a linear layer is used to project the fused features onto a lower-dimensional space for prediction. Experimental results show that SA-CMD outperforms existing baseline models in the public CMU-MOSI and CMU-MOSEI datasets across various evaluation metrics, especially in terms of the Pearson correlation coefficient, binary classification accuracy, and seven-class classification accuracy. Ablation experiments further verify the key role of the self-attention mechanism and the center moment discrepancy method in enhancing model performance, fully demonstrating the effectiveness and robustness of the SA-CMD model in multimodal sentiment analysis tasks.

关键词

多模态情感分析 / 自注意力机制 / 中心距差异 / 多模态特征融合

Key words

multimodal sentiment analysis / self-attention mechanism / center moment discrepancy / multimodal feature fusion

参考文献

[1] ZADEH A A B, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU—MOSEI dataset and interpretable dynamic fusion graph[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne: ACL, 2018: 2236-2246.

[2] TSAI Y H H, LIANG P P, ZADEH A, et al. Learning factorized multimodal representations[EB/OL]. (2018—06—16)[2024—07—04]. https://arxiv.org/abs/1806.06176.

[3] TSAI Y H H, BAI S, LIANG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]// Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: ACL, 2019: 6558-6569.

[4] LIU Weide, ZHAN Huijing, CHEN Hao, et al. Multimodal sentiment analysis with missing modality: a knowledge—transfer approach[EB/OL]. (2023—11—28)[2024—07—04]. https://arxiv.org/abs/2401.10747.

[5] 罗渊贻, 吴锐, 刘家锋, 等. 面向情感语义不一致的多模态情感分析方法[J/OL]. 计算机研究与发展.(2024—03—09)[2024—07—04]. http://kns.cnki.net/kcms/detail/11.1777.tp.20240305.1731.006.html.

[6] LUO Yuanyi, WU Rui, LIU Jiafeng, et al. Multimodal sentiment analysis method for sentimental semantic inconsistency[J/OL]. Journal of Computer Research and Development. (2024—03—09)[2024—07—04]. http://kns.cnki.net/kcms/detail/11.1777.tp.20240305.1731.006.html.

[7] BALTRUŠAITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(2): 423-443.

[8] SUN Z, SARMA P K, SETHARES W, et al. Multi—modal sentiment analysis using deep canonical correlation analysis[EB/OL]. (2019—07—15)[2024—07—04]. https://arxiv.org/abs/1907.08696.

[9] MAI Sijie, HU Haifeng, XING Songlong. Modality to modality translation: an adversarial representation learning and graph fusion network for multimodal fusion[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI Press, 2020: 164-172.

[10] HAZARIKA D, ZIMMERMANN R, PORIA S. Misa: modality—invariant and—specific representations for multimodal sentiment analysis[C]// Proceedings of the 28th ACM International Conference on Multimedia. Seattle: ACM, 2020: 1122-1131.

[11] ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[EB/OL]. (2017—07—23)[2024—07—04]. https://arxiv.org/abs/1707.07250.

[12] LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low—rank multimodal fusion with modality—specific factors[EB/OL]. (2018—05—31)[2024—07—04]. https://arxiv.org/abs/1806.00064.

[13] SUN Z, SARMA P, SETHARES W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Honolulu: AAAI Press, 2020: 8992-8999.

[14] HAN W, CHEN H, PORIA S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[EB/OL]. (2019—09—01)[2024—07—04]. https://arxiv.org/abs/2109.00412.

[15] 田昌宁, 贺昱政, 王笛, 等. 基于Transformer的多子空间多模态情感分析[J/OL]. 西北大学学报(自然科学版). (2024—04—03)[ 2024—07—04]. https://xdxbzk.nwu.edu.cn/thesisDetails#10.16152/j.cnki.xdxzr.2024—02—002&lang=zh.

[16] TIAN Changning, HE Yuzheng, WANG Di, et al. Multi—subspace multimodal sentiment analysis method based on Transformer[J/OL]. Journal of Northwest University (Natural Science Edition). (2024—04—03)[ 2024—07—04]. https://xdxbzk.nwu.edu.cn/thesisDetails#10.16152/j.cnki.xdxzr.2024—02—002&lang=zh.

[17] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL]. (2014—09—03)[2024—07—04]. https://arxiv.org/abs/1409.0473.

[18] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.

[19] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre—training of deep bidirectional transformers for language understanding[EB/OL]. (2018—10—11)[2024—07—04]. https://arxiv.org/abs/1810.04805.

[20] WANG Di, GUO Xutong, TIAN Yumin, et al. TETFN: a text enhanced transformer fusion network for multimodal sentiment analysis[J]. Pattern Recognition, 2023, 136: 109259.

[21] HOCHREITER S, SCHMIDHUBER J. Long short—term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.

[22] ZELLINGER W, GRUBINGER T, LUGHOFER E, et al. Central moment discrepancy (CMD) for domain—invariant representation learning[EB/OL]. (2017—02—28)[2024—07—04]. https://arxiv.org/abs/1702.08811.

[23] YU Weimeng, XU Hua, YUAN Ziqi, et al. Learning modality—specific representations with self—supervised multi—task learning for multimodal sentiment analysis[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Virtual: AAAI Press, 2021, 35(12): 10790-10797.

[24] ZADEH A, ZELLERS R, PINCUS E, et al. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[EB/OL]. (2016—06—20)[2024—07—04]. https://arxiv.org/abs/1606.06259.

[25] LIN Han, ZHANG Pinglu, LIN Jiading, et al. PS—mixer: a polar—vector and strength—vector mixer model for multimodal sentiment analysis[J]. Information Processing & Management, 2023, 60(2): 103229.

[26] FAN Qinglan, ZHANG Yunfeng, BAO Fangxun, et al. An region adaptive image interpolation algorithm based on the NSCT[J]. Journal of Computer Research and Development, 2018, 55(3): 629-642.

[27] DU C J, SUN D W. Recent developments in the applications of image processing techniques for food quality evaluation[J]. Trends in Food Science & Technology, 2004, 15(5): 230-249.

[28] MENDOZA F, DEJMEK P, AGUILERA J M. Colour and image texture analysis in classification of commercial potato chips[J]. Food Research International, 2007, 40(9): 1146-1154.

[29] LEE K C, HO S J, HO S Y. Accurate estimation of surface roughness from texture features of the surface image using an adaptive neuro—fuzzy inference system[J]. Precision Engineering, 2005, 29(1): 95-100.

[30] YAO X X, WU Q, ZHANG P, et al. Weighted adaptive image super—resolution scheme based on local fractal feature and image roughness[J]. IEEE Transactions on Multimedia, 2021, 23: 1426-1441.

[31] CHEN Z Y, MURAMATSU S, ABE Y. Fast image super—resolution via multiple directional transforms[C]// 2016 IEEE International Conference on Image Processing (ICIP). Phoenix: IEEE, 2016: 1434-1438.

[32] FRITSCH E M, GU S, TIMOFTE R. Frequency separation for real—world super—resolution[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul: IEEE, 2019: 3599-3608.

[33] DA CUNHA A L, ZHOU J, DO M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10): 3089-3101.

[34] BARNSLEY M F. Fractal functions and interpolation[J]. Constructive Approximation, 1986, 2(1): 303-329.

[35] MANDELBROT B B. The fractal geometry of nature[M]. New York: W H Freeman and Company, 1982: 131-147.

[36] BOUDA M, CAPLAN J S, SAIERS J E. Box—counting dimension revisited: presenting an efficient method of minimizing quantization error and an assessment of the self—similarity of structural root systems[J]. Frontiers in Plant Science, 2016, 7: 149.

[37] BAO F X, YAO X X, SUN Q H, et al. Smooth fractal surfaces derived from bicubic rational fractal interpolation functions[J]. Science China Information Sciences, 2018, 61(9): 099104.

[38] EBERHART, SHI Y H. Particle swarm optimization: developments, applications and resources[C]// Proceedings of the 2001 Congress on Evolutionary Computation. Seoul: IEEE, 2002: 81-86.

[39] TIMOFTE R, DE V, VAN GOOL L. Anchored neighborhood regression for fast example—based super—resolution[C]// 2013 IEEE International Conference on Computer Vision. Sydney: IEEE, 2014: 1920-1927.

[40] DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super—resolution[C]// Computer Vision—ECCV 2014. Cham: Springer, 2014: 184-199.

[41] LAI W S, HUANG J B, AHUJA N, et al. Fast and accurate image super—resolution with deep Laplacian pyramid networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613.

[42] SHOCHER A, COHEN N, IRANI M. Zero—shot super—resolution using deep internal learning[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3118-3126.

[43] ZEYDE R, ELAD M, PROTTER M. On single image scale—up using sparse—representations[C]// Curves and Surfaces. Berlin, Heidelberg: Springer, 2012: 711-730.

[44] FRANZEN R. Kodak lossless true color image suite[DS/OL]. (2024—10—25)[2026—01—14]. https://doi.org/10.48550/arXiv.2402.16855.

基金资助

国家自然科学基金资助项目(62166039)