基于文本特征对齐的多模态视频异常检测

Multimodal Video Anomaly Detection Based on Textual Feature Alignment

摘要

弱监督多模态视频异常检测旨在仅利用视频级标签，通过融合视觉、音频等多模态数据，实现对视频中异常事件片段的精确识别与定位，是计算机视觉领域一项极具挑战性的任务。当前方法面临两大瓶颈：一是跨模态时序相关性建模不足，限制了异常定位的精度；二是文本提示模块多依赖固定语义，缺乏动态上下文感知，导致语义引导能力有限。为解决上述问题，提出一种基于文本特征对齐的多模态视频异常检测方法(multimodal video anomaly detection based on textual feature alignment，TFA-MVAD)。该方法首先设计了一个跨模态特征融合模块，通过动态注意力机制捕捉音视频片段间的时序依赖关系，实现多模态互补信息的深度整合，从而提升关键片段的定位能力。其次，引入特征对齐模块，将预训练文本语义特征与融合后的音视频特征进行对齐，并借助对比学习策略增强跨模态语义一致性。实验结果表明，该方法在 XD-Violence 数据集上表现优异，与现有最佳方法相比，其平均精度(Average Precision，AP)提升了 3.21%，验证了该方法的有效性。

Abstract

Weakly supervised multi-modal video anomaly detection aims to accurately identify and localize anomalous event segments in videos by effectively integrating multi-modal data such as visual and audio information using only video-level annotations. It is a highly challenging task in the field of computer vision. On one hand, insufficient modeling of cross-modal temporal correlations limits the precision of anomaly event localization. On the other hand, the text prompt module relies on fixed semantic information and lacks dynamic contextual associations, which constrains the effectiveness of semantic guidance. To address these challenges, this paper proposes a multi-modal video anomaly detection method based on textual feature alignment (TFA-MVAD). The method first designs a cross-modal feature fusion module, which models the temporal correlations between audio-visual segments through a dynamic attention mechanism, achieving deep integration of complementary multi-modal information and significantly improving the localization capability of key segments. Secondly, a feature alignment module is introduced to align pre-trained textual semantic features with the fused audio-visual features, optimizing cross-modal semantic consistency through a contrastive learning strategy. Experimental results demonstrate that the proposed method achieves significant performance improvements on the XD-Violence dataset. Compared to the state-of-the-art methods, it increases the Average Precision (AP) by 3.21%, validating the effectiveness of the approach.

关键词

多模态融合 / 文本信息 / 特征对齐 / 时序相关性

Key words

multimodal fusion / textual information / feature alignment / temporal correlation

参考文献

[1] 汪洋，周脚根，严俊，等. 基于深度学习的监控视频异常检测方法综述[J].中国图像图形学报，2025，30(3):615-640.

[2] Cheng B，Wenjia Z，Sun X，et al. A novel spatio—temporal information knowledge distillation strategy for weak—supervised video anomaly detection[C]//The 7th International Conference on Pattern Recognition and Artificial Intelligence，Hangzhou，China，2024:398-403.

[3] 李歌，肖洪兵，闫善武，等. 基于多尺度时空特征融合的视频异常事件检测[J].燕山大学学报，2025，49(1):74-82.

[4] 余烨，陈维笑，陈凤欣. 面向车型识别的夜间车辆图像增强网络 RIC—NVNet[J].中国图像图形学报，2023，28(7):2054-2067.

[5] Zhou X，Peng X，Luo Y，et al. Learning weakly supervised audio—visual violence detection in hyperbolic space[J].Image and Vision Computing，2024，151:105286.

[6] Chen J，Li L，Su L，et al. Prompt—enhanced multiple instance learning for weakly supervised video anomaly detection[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，WA，USA，2024:18319-18329.

[7] Wu P，Liu J，Shi Y，et al. Not only look，but also listen: Learning multimodal violence detection under weak supervision[C]//16th European Conference on Computer Vision. Glasgow，UK，2020:322-339.

[8] Pang W F，He Q H，Hu Y，et al. Violence detection in videos based on fusing visual and audio information[C]//ICASSP 2021—2021 IEEE international conference on acoustics，speech and signal processing (ICASSP). Toronto，Ontario，Canada，2021:2260-2264.

[9] Pu Y，Wu X. Audio—guided attention network for weakly supervised violence detection[C]//2022 2nd International Conference on Consumer Electronics and Computer Engineering (ICCECE). Guangzhou，China，2022:219-223.

[10] Sun S，Gong X. Multi—scale bottleneck transformer for weakly supervised multimodal violence detection[C]//IEEE International Conference on Multimedia and Expo (ICME). Niagara Falls，Canada，2024:1-6.

[11] Yu J，Liu J，Cheng Y，et al. Modality—aware contrastive instance learning with self—distillation for weakly—supervised audio—visual violence detection[C]//Proceedings of the 30th ACM International Conference on Multimedia. Lisbon，Portugal，2022:6278-6287.

[12] Pu Y，Wu X，Yang L，et al. Learning prompt—enhanced context features for weakly—supervised video anomaly detection[J].IEEE Transactions on Image Processing，2024，33:4923-4936.

[13] Zhou X，Peng X，Wen H，et al. Learning weakly supervised audio—visual violence detection in hyperbolic space[J].Image and Vision Computing，2024，151:105286.

[14] Ghadiya A，Kar P，Chudasama V，et al. Cross—modal fusion and attention mechanism for weakly supervised video anomaly detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，WA，USA，2024:1965-1974.

[15] Sharma M K，Sheet D，Biswas P K. Local instance and context dictionary—based detection and localization of abnormalities[J].Machine Vision and Applications，2021，32(3):69.

[16] Speer R，Havasi C. Representing general relational knowledge in ConceptNet 5[J].European Language Resources Association (ELRA)，2012.

[17] 朱文佳，张婷，程茹秋，等. 考虑背景失真的无参考视频质量评价方法[J].南京师大学报(自然科学版)，2025，48(3):102-111.

[18] Zhang C，Li G，Qi Y，et al. Exploiting completeness and uncertainty of pseudo labels for weakly supervised video anomaly detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver，BC，Canada，2023:16271-16280.

[19] Zhou H，Yu J，Yang W. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Washington，DC，USA，2023，37(3):3769-3777.

[20] Wu P，Liu X，Liu J. Weakly supervised audio—visual violence detection[J].IEEE Transactions on Multimedia，2023，25:1674-1685.

[21] Al Marri S，Zaheer M Z，Nandakumar K. A multi—head approach with shuffled segments for weakly—supervised video anomaly detection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa，HI，USA，2024:132-142.

[22] Meng J，Tian H，Lin G，et al. Audio—visual collaborative learning for weakly supervised video anomaly detection[J].IEEE Transactions on Multimedia，2025，27:1-12.

[23] Wu P，Liu J，He X，et al. Toward video anomaly retrieval from video anomaly detection: new benchmarks and model[J].IEEE Transactions on Image Processing，2024，33:2213-2225.

基金资助

国家自然科学基金面上资助项目(62372153)

安徽省自然科学基金面上资助项目(2308085MF216)