基于 GainPose的学生课堂行为识别方法

A student classroom behavior recognition method based on GainPose

摘要

针对真实课堂场景中学生分布密集、遮挡严重导致的行为识别困难问题,提出了一种基于 GainPose的学生课堂行为识别方法:使用嵌入通道空间注意力模块 CBAM (convolutional block attention module)和小目标检测模块的改进 YOLOv7作为人体检测器,实现学生的精准定位;以基于关键点缺失数据预测的 GainPose作为姿态估计器,融合生成对抗式插补网络(generative adversarial imputation network, Gain)对缺失的关键点数据进行预测,获取完整的学生上半身关键点信息;采用轻量级 MobileNetV3 作为单人姿态估计器(single person pose estimator, SPPE)的骨干网络,以减少网络计算量;根据课堂场景,将时空图卷积网络的空间图卷积策略调整为更适配的空间配置分区策略,用于学生课堂行为分类.验证实验结果表明,经 TensorRT 加速后,该方法在 GPU 上的单幅图像推理时间为1.97 ms,检测速度为95.67 f/s,平均精度均值达96.79%,能够胜任学生课堂行为的实时检测任务.

Abstract

To address the difficulty of behavior recognition caused by dense distribution and severe occlusion of students in real classroom scenarios, this paper proposes a student classroom behavior recognition method based on GainPose. An improved YOLOv7 embedded with the convolutional block attention module (CBAM) and a small object detection module is used as the human detector to achieve accurate student localization. GainPose, based on missing keypoint data prediction, is employed as the pose estimator, integrating the generative adversarial imputation network (Gain) to predict missing keypoint data, thereby obtaining complete upper-body keypoint information of students. A lightweight MobileNetV3 is adopted as the backbone network of the single person pose estimator (SPPE) to reduce computational complexity. According to the classroom scenario, the spatial graph convolution strategy of the spatial temporal graph convolutional network is adjusted to a more adaptive spatial configuration partitioning strategy for student classroom behavior classification. Experimental results show that after acceleration with TensorRT, the proposed method achieves an inference time of 1.97 ms per image on GPU, a detection speed of 95.67 f/s, and a mean average precision of 96.79%, demonstrating its capability for real-time detection of student classroom behaviors.

关键词

课堂行为 / 行为识别 / 目标检测 / 姿态估计 / 时空图卷积

Key words

classroom behavior / behavior recognition / object detection / pose estimation / spatial temporal graph convolution

参考文献

[1] 中华人民共和国教育部. 教育部关于印发《高等学校人工智能创新行动计划》的通知[EB/OL].(2018-04-03)[2025-01-25]. http://www.moe.gov.cn/srcsite/A16/s7062/201804/t20180410_332722.html.

[2] 孙众, 吕恺悦, 骆力明, 等. 基于人工智能的课堂教学分析[J]. 中国电化教育, 2020(10): 15-23.

[3] 蔡博深, 赵玲, 周容, 等. 基于图卷积神经网络的学生课堂不规范行为实时检测方法[J]. 系统仿真技术, 2022, 18(2): 141-146.

[4] 刘宏, 张释文, 孙程, 等. 基于视频时空特征的学生课堂行为检测[J]. 软件导刊, 2024(8): 267-274.

[5] LI L N, LIU M H, SUN L Y, et al. ET-YOLOv5s: toward deep identification of students' in-class behaviors[J]. IEEE Access, 2022, 10: 44200-44211.

[6] 董琪琪. 基于目标检测算法的学生课堂行为识别研究[D].天津:河北工业大学, 2020.

[7] 周宇航, 张海宇, 王永彬, 等. 基于 AlphaPose的课堂行为识别算法[J]. 信息技术与信息化, 2023(12): 204-208.

[8] 张志强. 基于轻量级网络的人体姿态估计算法研究[D].成都:电子科技大学, 2022.

[9] 王泽杰, 沈超敏, 赵春, 等. 融合人体姿态估计和目标检测的学生课堂行为识别[J]. 华东师范大学学报(自然科学版), 2022(2): 55-66.

[10] VIÑAS R, AZEVEDO T, GAMAZO N E R, et al. Gene expression imputation with generative adversarial imputation nets[EB/OL]. [2025-01-25]. https://doi.org/10.1101/2020.06.09.141689.

[11] HOWARD A, SANDLER M, CHEN B, et al. Searching for MobileNetV3[C]// 2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 1314-1324.

[12] WU L M, WANG Y F. Flame detection based on improved YOLOv7[J]. Journal of Physics: Conference Series, 2023, 2589(1): 012006.

[13] 贾轶钧, 杨辉跃. 基于计算机视觉的课堂行为识别方法综述[J]. 自动化与仪器仪表, 2022(9): 1-6.

[14] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]// AAAI Conference on Artificial Intelligence. Louisiana: AAAI, 2018: 7444-7452.

[15] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// 2018 European Conference on Computer Vision. Cham: Springer, 2018: 3-19.

[16] 贾雪莹, 赵春江, 周娟, 等. 基于改进 YOLOv7模型的柑橘表面缺陷在线检测[J]. 农业工程学报, 2023, 39(23): 142-151.

[17] CAO Z, SIMON T, WEI S H, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1302-1310.

[18] ZHOU W J, LIU J F, LEI J S, et al. GMNet: graded-feature multilabel-learning network for RGB-thermal urban scene semantic segmentation[J]. IEEE Transactions on Image Processing, 2021, 30: 7790-7802.

[19] CHENG B W, XIAO B, WANG J D, et al. HigherHRNet: scale-aware representation learning for bottom-up human pose estimation[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 5385-5394.

基金资助

黑龙江省高等教育教学改革研究项目(SJGYB2024440)