基于终端位姿感知的多模态视频抽稀算法

Multimodal Video Sparsification Algorithm Based on Terminal Pose Perception

摘要

针对第一视角长时视频冗余信息多、传统抽帧难以捕获语义关键帧的问题，本文提出一种基于终端设备位姿感知的多模态大模型视频抽稀与分析算法。该方法充分利用同步获取的惯性测量单元数据、音频及视觉内容，实现多触发协同的关键帧抽取机制，通过位姿突变、语音语义触发和视觉显著性综合判定，精准捕获关键事件。在此基础上，将关键帧组织为语义片段，与位姿统计、语音转写等元信息绑定，形成适配大模型输入的“图像序列＋文本描述”结构。端侧执行轻量抽稀，云侧大模型进行推理分析，实现高效协同。在自建工业端子排巡检数据集上的实验结果表明，该方法在关键帧检测准确率、语义覆盖度、压缩比及实时性上均优于对比方法。该研究为长时第一视角视频智能分析提供了高效、可部署的工程化方案。

Abstract

To address the issue that long first-person videos contain a lot of redundant information and traditional frame sampling struggles to capture key semantic moments, this paper proposes a multimodal large-model-based video sparsification and analysis method driven by terminal device pose awareness. The method makes full use of synchronously collected data, including inertial measurement unit data, audio and visual content. It uses a multi-trigger collaborative key frame extraction mechanism, combining sudden pose changes, speech-based semantic triggers and visual saliency to accurately capture important events. On top of that, the extracted key frames are organized into semantic segments, which are linked with metadata such as pose statistics and speech transcriptions. These are then structured into a "image sequence+text description" format suitable for large model input. The system performs lightweight video thinning on the device side, while the cloud-based large model handles inference and analysis, enabling efficient collaboration. Experimental results on a self-built industrial terminal block inspection dataset show that this method outperforms baseline approaches in key frame detection accuracy, semantic coverage, compression ratio, and real-time performance. This study provides an efficient and deployable engineering solution for intelligent analysis of long-duration first-person videos.

关键词

视频抽稀 / 多模态大模型 / 终端位姿感知 / 第一视角视频

Key words

video sparsification / multimodal large model / terminal pose perception / first-person video

参考文献

[1] Sahu A, Chowdhury A S. Summarizing egocentric videos using deep features and optimal clustering[J].Neurocomputing, 2020, 398:209-221.

[2] Sahu A, Chowdhury A S. Egocentric video co—summarization using transfer learning and refined random walk on a constrained graph[J].Pattern Recognition, 2023, 134:109128.

[3] Hong F, Guzov V, Kim H J, et al. EgoLM:multi—modal language model of egocentric motions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2025:5344-5354.

[4] Moussoui H, Akkad N, Benslimane M. A review of video summarization[C]//Proceedings of the 3rd International Conference on Digital Technologies and Applications. Fez, Morocco:Springer, 2023:516-525.

[5] Zhao B, Gong M, Li X. Hierarchical multimodal transformer to summarize videos[J].Neurocomputing, 2022, 468:360-369.

[6] Patel A, Chitalia V, Yang Y. Advancing egocentric video question answering with multimodal large language models[J].arXiv Preprint, arXiv:2504.04550, 2025.

[7] Xie J, Chen X, Zhao S, Lu S P. Video summarization via knowledge—aware multimodal deep networks[J].Knowledge—based Systems, 2024, 293:111670.

[8] Tang X, Qiu J, Xie L, et al. Adaptive key frame sampling for long video understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2025:29118-29128.

[9] Lee M J, Gong D, Cho M. Video summarization with large language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2025:18981-18991.

[10] Meena P, Kumar H, Yadav S K. A review on video summarization techniques[J].Engineering Applications of Artificial Intelligence, 2023, 118:105667.

[11] Tan K, Zhou Y, Xia Q, et al. Large model based sequential key frame extraction for video summarization[J].arXiv Preprint, arXiv:2401.04962, 2024.

[12] Cakmak M C, Agarwal N, Poudel D. TriPSS:a tri—modal key frame extraction framework using perceptual, structural, and semantic representations[J].arXiv Preprint, arXiv:2506.05395, 2025.

[13] Li W, Qi D, Zhang C, et al. Video summarization based on mutual information and entropy sliding window method[J].Entropy, 2020, 22(11):1285.

[14] Ranganath V, Padmavathi S, Aarthi R. Optimal frame extraction for event detection in surveillance videos[J].Procedia Computer Science, 2025, 256:1192-1199.

[15] Arslan S, Tanberk S. Key frame extraction with attention based deep neural networks[J].arXiv Preprint, arXiv:2306.13176, 2023.

[16] Kudo K, Nagasawa H, Suzuki J, et al. A challenging multimodal video summary:simultaneously extracting and generating key frame—caption pairs from video[J].arXiv Preprint, arXiv:2312.01575, 2023.

[17] 韩乾乾，顾华宁，金湓，等 。基于 X3D 特征和语义融合的篮球运动员检测跟踪方法[J].南京师大学报(自然科学版)，2025，48(6):101-110.

[18] 何俊，张彩庆，李小珍，等 。面向深度学习的多模态融合技术研究综述[J].计算机工程，2020，46(5):1-11.

基金资助

国家自然科学基金资助项目(62503220)

江苏省自然科学基金资助项目(BK20230706)