抑郁症识别中的动态加权多模态融合框架

Dynamic Weighted Multimodal Fusion Framework in Depression Identification

摘要

针对现有抑郁症单模态识别方法(如 EEG(Electroencephalogram)、面部表情、文本分析)易受环境干扰且识别率有限的问题, 提出了 MFF-DR-DWA(Multimodal Fusion Framework for Depression Recognition with Dynamic Weight Adjustment)多模态融合框架。其通过动态整合脑电信号、语言特征和面部表情3种异质数据源, 并采用自适应加权算法优化各模态贡献度。实验结果表明, 在标准测试数据集上, 该模型的准确率为 99.09%、F₁值为 99.12% 和 AUC(Area Under Curve)值为 99.97%, 其性能显著优于传统单模态分析方法和固定权重融合方案。该多模态融合策略有效提升了抑郁症识别的准确性和鲁棒性。

Abstract

To address the limitations of existing unimodal recognition methods for depression (e.g., EEG (Electroencephalogram), facial expressions, text analysis), which are susceptible to environmental noise and exhibit limited accuracy, an innovative MFF-DR-DWA(Multimodal Fusion Framework is proposed for Depression Recognition with Dynamic Weight Adjustment). The framework dynamically integrates three heterogeneous data modalities-electroencephalography (EEG), linguistic features, and facial expressions-while employing an adaptive weighting algorithm to optimize the contribution of each modality. Experimental results on benchmark datasets demonstrate that the proposed model achieves an accuracy of 99.09%, an F ₁-score of 99.12%, and an AUC(Area Under Curve) of 99.97%, significantly outperforming conventional unimodal approaches and static fusion methods. This multimodal fusion strategy effectively enhances the accuracy and robustness of depression recognition.

关键词

多模态 / 动态加权 / 多模态融合 / 抑郁症识别 / 注意力机制

Key words

multimodal / dynamic weighting / multimodal fusion / depression recognition / attention mechanism

参考文献

[1] 马思梦, 曹勇, 王培琳, 等. 深度学习算法在面向行为分析的抑郁症辅助诊断中的研究进展[J]. 中华精神科杂志, 2020, 53(5): 460-463.

[2] MA S M, CAO Y, WANG P L, et al. Research Progress of Deep Learning Algorithms in the Auxiliary Diagnosis of Depression Based on Behavior Analysis[J]. Chinese Journal of Psychiatry, 2020, 53(5): 460-463.

[3] 张鑫衍. 基于文本挖掘的社交网络用户精神疾病筛查[J]. 现代信息科技, 2023, 7(15): 157-161.

[4] ZHANG X Y. Social Network User Mental Disease Screening Based on Text Mining[J]. Modern Information Technology, 2023, 7(15): 157-161.

[5] 姜月武, 路东生, 党良慧, 等. 人脸表情识别研究进展[J]. 智能计算机与应用, 2021, 11(6): 43-50.

[6] JIANG Y W, LU D S, DANG L H, et al. Research Progress on Facial Expression Recognition[J]. Intelligent Computer and Applications, 2021, 11(6): 43-50.

[7] 刘峻杉. 基于多模态数据的青少年抑郁症诊断模型关键技术研究[D]. 成都: 四川大学, 2021.

[8] LIU J S. Research on Key Technologies of Adolescent Depression Diagnosis Model Based on Multimodal Data[D]. Chengdu: Sichuan University, 2021.

[9] 朱永新. “五育并举”构建一体化青少年心理健康教育体系 关于青少年心理健康教育问题的探讨[J]. 苏州大学学报(教育科学版), 2025, 13(1): 1-12.

[10] ZHU Y X. Constructing an Integrated Adolescent Mental Health Education System through the “Five Education”Approach: A Discussion on Adolescent Mental Health Education Issues[J]. Journal of Soochow University (Education Science Edition), 2025, 13(1): 1-12.

[11] 李宸芮, 吴毅明, 陈新旺, 等. 脑电静息微状态在广泛性焦虑症中的应用与研究进展[J]. 中国比较医学杂志, 2025, 35(2): 158-164.

[12] LI C R, WU Y M, CHEN X W, et al. Application and Research Progress of EEG Resting Microstate in Generalized Anxiety Disorder[J]. Chinese Journal of Comparative Medicine, 2025, 35(2): 158-164.

[13] FARRUQUE N, GOEBEL R, SIVAPALAN S, et al. Deep Temporal Modelling of Clinical Depression through Social Media Text [J/OL]. Natural Language Processing Journal, 2024, 6: 100052 (2024-03-01) [2025-03-23]. https://www.sciencedirect.com/science/article/pii/S2949719123000493.

[14] WANG S, WANG X, WANG S, et al. Bi-Directional Long Short-Term Memory Method Based on Attention Mechanism and Rolling Update for Short-Term Load Forecasting[J]. International Journal of Electrical Power & Energy Systems, 2019, 109: 470-479.

[15] WU Y, LIU Z, YUAN J, et al. PIE: A Personalized Information Embedded Model for Text-Based Depression Detection [J/OL]. Information Processing & Management, 2024, 61(6): 103830 (2024-11-01) [2025-08-23]. https://www.sciencedirect.com/science/article/pii/S0306457324001894.

[16] DING Z, HU Y, JING R, et al. A Depression Recognition Method Based on the Alteration of Video Temporal Angle Features [J/OL]. Applied Sciences, 2023, 13(16): 9230 (2023-08-14) [2025-03-23]. https://www.mdpi.com/2076-3417/13/16/9230.

[17] HAN K, WANG Y H, TIAN Q, et al. GhostNet: More Features from Cheap Operations[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE/CVF, 2020: 1580-1589.

[18] LI X Y, YI X Y, YE J Y, et al. SFTNet: A Microexpression-Based Method for Depression Detection [J/OL]. Computer Methods and Programs in Biomedicine, 2023, 243: 107923 [2025-05-10]. https://doi.org/10.1016/j.cmpb.2023.107923.

[19] AHMED N N, BHAT T K, POWAR O S. Stacked Ensemble Machine Learning Approach for Electroencephalography Based Major Depressive Disorder Classification Using Temporal Statistics [J/OL]. Systems Science & Control Engineering, 2024, 12(1): 2427028 (2024-07-04)[2025-03-23]. https://www.tandfonline.com/doi/full/10.1080/21642583.2024.2427028.

[20] XU X, XU J, DU R, et al. Research on Depression Recognition Model and Its Temporal Characteristics Based on Multiscale Entropy of EEG Signals [J/OL]. Entropy, 2025, 27(2): 142 [ 2025-03-23]. https://pubmed.ncbi.nlm.nih.gov/40003139/.

[21] HE M, BAKKER M E, LEW S M. DPD (DePression Detection) Net: A Deep Neural Network for Multimodal Depression Detection [J/OL]. Health Information Science and Systems, 2024, 12(1): 53 (2024-11- 12)[2025-03-23]. https://link.springer.com/article/10.1007/s13755-024-00311-9.

[22] CHENG Z, HUANG X, DING Y. An Intelligent Depression Detection Model Based on Multimodal Fusion Technology [J/OL]. Journal of Mechanics in Medicine and Biology, 2024, 24(8) (2024-08-22) [2025-03-23]. https://www.worldscientific.com/doi/full/10.1142/S0219519424400463.

[23] MOHAMMAD F, ANSOOR A M K. MDD: A Unified Multimodal Deep Learning Approach for Depression Diagnosis Based on Text and Audio Speech[J]. Computers, Materials & Continua, 2024, 81(3): 4125-4147.

[24] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer[J]. Journal of Machine Learning Research, 2020, 21: 5485-5551.

[25] 赵小明, 范慧婷, 张石清. 一种基于多模态特征增强网络的抑郁症检测方法[J]. 软件工程, 2024, 27(10): 68-73.

[26] ZHAO X M, FAN H T, ZHANG S Q. A Depression Detection Method Based on Multimodal Feature Enhancement Network[J]. Software Engineering, 2024, 27(10): 68-73.

[27] MUMTAZ W, XIA L, YASIN M A M, et al. A Wavelet-Based Technique to Predict Treatment Outcome for Major Depressive Disorder [J/OL]. PLoS ONE, 2017, 12(2): e0171409 (2017-02-02) [2025-03-23]. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0171409.

[28] KUMAR S J, BHUVANESWARI P. Analysis of Electroencephalography (EEG) Signals and Its Categorization: A Study[J]. Procedia Engineering, 2012, 38: 2525-2536.

[29] LIU Y, OTT M, GOYAL N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach [EB/OL]. (2019-07-26) [2025-03-23]. Available: https://arxiv.org/abs/1907.11692.

[30] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA, USA: Association for Computational Linguistics, 2019: 4171-4186.

[31] HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, New Jersey, USA: IEEE, 2016: 770-778.

[32] WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional Block Attention Module[C]// Proceedings of the European Conference on Computer Vision (ECCV). Cham: Springer, 2018: 3-19.

[33] RAFIEI A, ZAHEDIFAR R, SITAULA C, et al. Automated Detection of Major Depressive Disorder with EEG Signals: A Time Series Classification Using Deep Learning[J]. IEEE Access, 2022, 10: 73804-73817.

[34] XIA M, ZHANG Y S, WU Y H, et al. An End-to-End Deep Learning Model for EEG-Based Major Depressive Disorder Classification[J]. IEEE Access, 2023, 11: 41337-41347.

[35] YANG Z, DAI Z, YANG Y, et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding[C]// Advances in Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2020: 5730-5740.

[36] SANH V, DEBUT L, CHAUMOND J, et al. DistilBERT: A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter [DB/OL]. (2019-10-02)[2025-03-23]. https://arxiv.org/abs/1910.01108.

[37] ZHANG P, ZENG G, WANG T, et al. TinyLlama: An Open-Source Small Language Model [DB/OL]. (2024-01-04)[2025-06-05]. https://arxiv.org/abs/2401.02385.

[38] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Piscataway, New Jersey, USA: IEEE, 2017: 2261-2269.

[39] SZEGEDY C, LIU W, JIA Y, et al. Going Deeper with Convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Piscataway, New Jersey, USA: IEEE Computer Society, 2015: 1-9.

[40] SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, New Jersey, USA: IEEE, 2018: 4510-4520.

[41] QIANG B H, ZHAI Y J, ZHOU M L, et al. SqueezeNet and Fusion Network-Based Accurate Fast Fully Convolutional Network for Hand Detection and Gesture Recognition[J]. IEEE Access, 2021, 9: 77661-77674.

[42] WOO S, DEBNATH S, HU R, et al. ConvNeXtV2: Co-Designing and Scaling ConvNets with Masked Autoencoders[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023). Piscataway, New Jersey, USA: IEEE Computer Society, 2023: 16133-16142.