基于多模态特征融合的视频暴力行为识别方法

Video Violence Recognition Method Based on Multi-modal Feature Fusion

摘要

视频暴力行为识别在智能安防领域发挥着重要作用. 现有方法大多使用单一模态，提供的信息有限，难以充分表达复杂的暴力行为，在实际场景下的适用性较低. 对此，本文提出了一种基于多模态特征融合的视频暴力行为识别方法，通过融合 RGB 视频帧提供的人体姿态信息、RGB 帧差提供的人体运动信息以及深度图提供的人物间相对位置信息，增强了暴力行为的特征表示，使其能够有效应对人群打斗中的遮挡现象和开放场景的光照变化. 在特征融合阶段利用注意力机制，模型自适应地学习不同模态特征的重要性，动态分配不同模态特征的权重，从而提高真实环境下暴力行为识别的准确性和鲁棒性. 所提方法在 RWF-2000、Hockey 和 Movies 数据集上进行实验，识别准确率分别为 88.75%、99%和 100%. 结果表明，所提方法能够进行有效的暴力行为识别.

Abstract

Video violence recognition plays a significant role in the field of intelligent security. Most existing methods rely on a single modality, providing limited information and making it difficult to fully represent complex violent behaviors, thus having relatively low applicability in real-world scenarios. This paper proposes a video violence recognition method based on multi-modal feature fusion, which enhances the feature representation of violent behavior by integrating the human pose information provided by RGB frames, the human motion information provided by RGB frame differences, and the relative position information between characters provided by depth maps, so that it can effectively cope with the occlusion phenomenon in crowd fights and the lighting changes in open scenes. During the feature fusion process, attention mechanism is applied to enable the model to adaptively learn the importance of multi-modal features and dynamically allocate weights for different features, thereby improving the accuracy and robustness of violence recognition in real-world environments. The proposed method was tested on the RWF-2000, Hockey, and Movies datasets, achieving recognition accuracies of 88.75%, 99% and 100%, respectively. The results indicate that the proposed method can effectively identify violent behaviors.

关键词

视频暴力行为识别 / 视频理解 / 多模态特征融合 / 注意力机制

Key words

video violence recognition / video understanding / multi-modal feature fusion / attention mechanism

参考文献

[1] 王晓龙. 基于轨迹分析的暴力行为识别算法研究[D].上海:上海交通大学,2015.

[2] Senst T, Eiselein V, Kuhn A, et al. Crowd violence detection using global motion—compensated lagrangian features and scale—sensitive video—level representation[J].IEEE Transactions on Information Forensics and Security, 2017, 12(12):2945-2956.

[3] Chen D, Wactlar H, Gao C, et al. Recognition of aggressive human behavior using binary local motion descriptors[C]//30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Vancouver, BC, Canada:IEEE, 2008:5238-5241.

[4] Deb T, Arman A, Firoze A. Machine cognition of violence in videos using novel outlier—resistant VLAD[C]//17th IEEE International Conference on Machine Learning and Applications. Orlando, Florida, USA:IEEE, 2018:989-994.

[5] Sun Z, Ke Q, Rahmani H, et al. Human action recognition from various data modalities:A review[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3):3200-3225.

[6] Cheng M, Cai K, Li M. RWF—2000:a open large scale video database for violence detection[C]//25th International Conference on Pattern Recognition. Milan, Italy:IEEE, 2021:4183-4190.

[7] Islam Z, Rukonuzzaman M, Ahmed R. Efficient two stream network for violence detection using separable convolutional LSTM[C]//International Joint Conference on Neural Networks. Shenzhen, China:IEEE, 2021:1-8.

[8] Ilg E, Mayer N, Saikia T, et al. FlowNet 2.0:evolution of optical flow estimation with deep networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA:IEEE, 2017:1647-1655.

[9] Sudhakaran S, Lanz O. Learning to detect violent videos using convolutional long short—term memory[C]//14th IEEE International Conference on Advanced Video and Signal Based Surveillance. Lecce, Italy:IEEE, 2017:1-6.

[10] Su Y, Lin G, Zhu J, et al. Human interaction learning on 3D skeleton point clouds for video violence recognition[C]//16th European Conference on Computer Vision(ECCV). Glasgow, UK:Springer, 2020:74-90.

[11] Hu J, Shen L, Albanie S, et al. Squeeze—and—excitation networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8):2011-2023.

[12] Ding C, Fan S, Zhu M, et al. Violence detection in video by using 3D convolutional neural networks[C]//10th International Symposium on Visual Computing(ISVC). Las Vegas, NV, USA:Springer, 2014:551-558.

[13] Li J, Jiang X, Sun T, et al. Efficient violence detection using 3D convolutional neural networks[C]//16th IEEE International Conference on Advanced Video and Signal Based Surveillance. Taipei, China:IEEE, 2019:1-8.

[14] Hanson A, Pnvr K, Krishnagopal S, et al. Bidirectional convolutional LSTM for the detection of violence in videos[C]//Computer Vision—ECCV 2018 Workshops. Munich, Germany:Springer, 2019:280-295.

[15] Pfeuffer A, Dietmayer K. Separable convolutional LSTMs for faster video segmentation[C]//2019 IEEE Intelligent Transportation Systems Conference(ITSC). Auckland, New Zealand:IEEE, 2019:1072-1078.

[16] Guillermo G, Sanmiguel J. Human skeletons and change detection for efficient violence detection in surveillance videos[J].Computer Vision and Image Understanding, 2023, 233:103739.

[17] Eigen D, Puhrsch C, Fergus R. Depth map prediction from a single image using a multi—scale deep network[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal, Canada:Curran Associates, 2014:2366-2374.

[18] Ramamonjisoa M, Du Y, Lepetit V. Predicting sharp and accurate occlusion boundaries in monocular depth estimation using displacement fields[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, Washington, USA:IEEE, 2020:14648-14657.

[19] Garg R, Bg VK, Carneiro G, et al. Unsupervised cnn for single view depth estimation:geometry to the rescue[C]//14th European Conference on Computer Vision(ECCV). Amsterdam, The Netherlands:Springer, 2016:740-756.

[20] Godard C, MacAodh O, Firman M, et al. Digging into self—supervised monocular depth estimation[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul, South Korea:IEEE, 2019:3828-3838.

[21] Yang LH, Kang BY, Huang ZL, et al. Depth anything:unleashing the power of large—scale unlabeled data[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, WA, USA:IEEE, 2024:10371-10381.

[22] Krizhevsky A, Sutskever I, Hinton E. Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing System 25(NIPS 2012). NV, USA:Curran Associates, 2012:1106-1114.

[23] Sandler M, Howard A, Zhu M, et al. MobileNetV2:inverted residuals and linear bottlenecks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA:IEEE, 2018:4510-4520.

[24] Lin M, Chen Q, Yan S. Network in network[C]//2nd Learning Representations(ICLR). Banff, Canada:Springer, 2014.

[25] Nievas B, Suarez D, Sukthankar R, et al. Violence detection in video using computer vision techniques[C]//14th International Conference on Computer Analysis of Images and Patterns. Seville, Spain:Springer, 2011:332-339.

[26] Wang W, Dong S, Zou K, et al. A lightweight network for violence detection[C]//Proceedings of the 2022 5th International Conference on Image and Graphics Processing. New York, USA:Association for Computing Machinery, 2022:15-21.

[27] Vijekis R, Raudonis V, Dervinis G. Efficient violence detection in surveillance[J].Sensors, 2022, 22(6):2216.

[28] Honarjoo N, Abdari A, Mansouri A. Violence detection in compressed video[J].Multimedia Tools and Applications, 2024, 83(29):73703-73716.

基金资助

国家自然科学基金项目(41971343)