利用掩码自编码器预训练的热红外视频呼吸监测方法^*

A Method for Respiratory Monitoring in Thermal Infrared Video Using Masked Autoencoder Pretraining

摘要

目的 通过热红外视频构建时空图，结合自监督学习方法自动捕获呼吸引发的口鼻区域温度变化趋势，实现长期非接触式呼吸监测，为情绪、应激反应、呼吸系统疾病等早期预警提供技术支撑。方法 针对现有方法对热红外温度变化趋势表征能力不足的问题，提出两阶段自监督重建时空图-下游微调方法：首先通过仿射变换矩阵完成普通摄像头与热红外摄像头的空间配准，实现关键点跟踪并构建时空图；考虑到呼吸信号敏感区域的个体差异，先通过自监督学习挖掘深度表征，再对隐向量微调以适配下游监测任务，同时提出进出气量比与呼吸时长平均绝对误差两项新评估指标。结果 数据集实验结果表明，该两阶段训练方法可精准捕获呼吸信号趋势，端到端特征学习的波形拟合精度显著优于传统方法及现有流行模型，核心性能指标分别为：平均绝对误差0.07±0.02、均方根误差0.69±0.11、皮尔逊相关系数0.15±0.04、进出气量比0.40±0.12/0.26±0.05、呼吸时长平均绝对误差0.79±0.19/0.79±0.10。结论 基于掩码自编码器（masked autoencoder）的自监督预训练波形分析方法在呼吸监测中展现频域与时域双重优势；同时发现呼吸阶段脉搏信号的形态学差异，呼气阶段光电容积脉搏波（photoplethysmography, PPG）信号峰度显著降低、偏度减小。该发现为从PPG形态特征维度评估心肺耦合功能与自主神经调节提供了新视角。

Abstract

Objective By constructing spatiotemporal maps from thermal infrared videos and applying self-supervised learning methods, temperature change trends in the oral and nasal regions caused by breathing can be automatically detected, enabling long-term, non-contact breathing monitoring. This offers technical support for early warning of emotions, stress responses, and respiratory system diseases. Methods To address the limitations of existing methods in characterizing temperature variation trends in thermal infrared, a two-stage self-supervised reconstruction spatio-temporal graph and downstream fine-tuning method is proposed. First, spatial registration between the ordinary camera and the thermal infrared camera is achieved using an affine transformation matrix, enabling key point tracking and the construction of a spatio-temporal graph. Considering individual differences in the sensitive areas of the respiratory signal, deep representations are initially extracted through self-supervised learning, and the latent vectors are then fine-tuned to adapt to downstream monitoring tasks. Additionally, two new evaluation metrics are introduced: the average absolute error of the ratio of inhaled to exhaled gas volumes and the average absolute error of respiratory duration. Results The experimental results on the dataset show that this two-stage training method can accurately capture the trend of respiratory signals. The waveform fitting accuracy of end-to-end feature learning is significantly better than that of traditional methods and current popular models. The core performance indicators are as follows: mean absolute error 0.07 ± 0.02, root mean square error 0.69 ± 0.11, Pearson correlation coefficient 0.15 ± 0.04, ratio of inhalation to exhalation volume 0.40 ± 0.12/0.26 ± 0.05, and mean absolute error of respiratory duration 0.79 ± 0.19/0.79 ± 0.10. Conclusion The self-supervised pre-training waveform analysis method based on a masked autoencoder demonstrates advantages in both the frequency and time domains for respiratory monitoring. Additionally, morphological differences in the pulse signal during the respiratory phase were identified. The kurtosis of the photoplethysmography (PPG) signal significantly decreased, and the skewness was reduced during the exhalation phase. This finding offers a new perspective for evaluating cardiopulmonary coupling function and autonomic nerve regulation from the morphological feature dimension of PPG.

关键词

自编码器 / 呼吸频率 / 生理监测

Key words

Autoencoder / Respiratory rate / Monitoring, physiologic

参考文献

[1] 赵昕, 李婉姝, 周锦, 等. 阻塞性睡眠呼吸暂停与儿童行为问题的关联研究. 四川大学学报 (医学版), 2025, 56(5): 1373-1379. doi: 10.12182/20250960101.

[2] ZHAO X, LI W S, ZHOU J, et al. Association between obstructive sleep apnea and behavioral problems in children. J Sichuan Univ (Med Sci), 2025, 56(5): 1373-1379. doi: 10.12182/20250960101.

[3] 李小英, 张晨旭, 曾萍, 等. 阻塞性睡眠呼吸暂停综合征对无心脑血管疾病老年人夜间动态血压的影响. 四川大学学报(医学版), 2025, 56(2): 464-469. doi: 10.12182/20250360110.

[4] LI X Y, ZHANG C X, ZENG P, et al. Effect of obstructive sleep apnea syndrome on nocturnal ambulatory blood pressure monitoring results in older adults without cardiovascular or cerebrovascular diseases. J Sichuan Univ (Med Sci), 2025, 56(2): 464-469. doi: 10.12182/20250360110.

[5] 黄通达, 植良娥, 税荣春, 等. 成人阻塞性睡眠呼吸暂停相关性心血管事件的研究进展. 实用医院临床杂志, 2025, 22(1): 190-192. doi: 10.3969/j.issn.1672-6170.2025.01.042.

[6] HUANG T D, ZHI L E, SHUI R C, et al. Research progress of cardiovascular events associated with obstructive sleep apnea in adults. Practical Journal of Clinical Medicine, 2025, 22(1): 190-192. doi: 10.3969/j.issn.1672-6170.2025.01.042.

[7] NAKAMURA N H, FUKUNAGA M, OKU Y. Respiratory modulation of cognitive performance during the retrieval process. PLoS One, 2018, 13(9): e0204021. doi: 10.1371/journal.pone.0204021.

[8] LIU S, GAO R X, FREEDSON P S. Non-invasive respiration and ventilation prediction using a single abdominal sensor belt// IEEE Signal Processing in Medicine and Biology Symposium, 2011: 1-5. doi: 10.1109/SPMB.2011.6120113.

[9] WANG T, SUN W, SHIMA K. Continuous body movement-robust respiration rate detection using frequency-modulated continuous wave millimeter wave radar based on non-negative matrix factorization algorithm. Biomedical Signal Processing and Control, 2025, 102: 107243. doi: 10.1016/j.bspc.2024.107243.

[10] TESTA O, CICCHETTI R, PISA S, et al. A novel FMCW radar scheme with millimeter motion detection capabilities suitable for cardio-respiratory monitoring. Sensors, 2025, 25(9): 2765. doi: 10.3390/s25092765.

[11] GIOIA F, PURA F, GRECO A, et al. Contactless estimation of respiratory frequency using 3D-CNN on thermal images. IEEE Journal of Biomedical and Health Informatics, 2025, 29(10): 7387-7395. doi: 10.1109/JBHI.2025.3567141.

[12] KHEDDAR H, HIMEUR Y, AMIRA A. BreathAI: transfer learning-based thermal imaging for automated breathing pattern recognition. IEEE International Conference on Image Processing (ICIP), 2025: 2612-2617. doi: 10.1109/ICIP55913.2025.11084681.

[13] ALVES R, Van MEULEN F, OVEREEM S, et al. Thermal cameras for continuous and contactless respiration monitoring. Sensors, 2024, 24(24): 8118. doi: 10.3390/s24248118.

[14] JAGADEV P, NAIK S, INDU GIRI L. Contactless monitoring of human respiration using infrared thermography and deep learning. Physiol Meas, 2022, 43(2): 025006. doi: 10.1088/1361-6579/ac57a8.

[15] VITAZKOVA D, FOLTAN E, KOSNACOVA H, et al. Advances in respiratory monitoring: a comprehensive review of wearable and remote technologies. Biosensors, 2024, 14(2): 90. doi: 10.3390/bios14020090.

[16] HE K, CHEN X, XIE S, et al. Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022: 16000-16009. doi: 10.1109/CVPR52688.2022.01553.

[17] LIU X, ZHANG Y, YU Z, et al. RPPG-mae: self-supervised pretraining with masked autoencoders for remote physiological measurements. IEEE Transactions on Multimedia, 2024, 26: 7278-7293. doi: 10.1109/TMM.2024.3363660.

[18] LUGARESI C, TANG J, NASH H, et al. Mediapipe: a framework for building perception pipelines. ArXiv, 2019: arXiv:1906.08172. doi: 10.48550/arXiv.1906.08172.

[19] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 770-778. doi: 10.1109/CVPR.2016.90.

[20] LIU Z, MAO H, WU C Y, et al. A ConvNet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 11976-11986. doi: 10.1109/CVPR52688.2022.01167.

[21] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale. ArXiv, 2021: arXiv:2010.11929. doi: 10.48550/arXiv.2010.11929.

[22] MAKOWSKI D, PHAM T, LAU Z J, et al. NeuroKit2: a Python toolbox for neurophysiological signal processing. Behav Res Methods, 2021, 53(4): 1689-1696. doi: 10.3758/s13428-020-01516-y.

[23] GALDI F, PEDONE C, MCGEE C A, et al. Inhaled high molecular weight hyaluronan ameliorates respiratory failure in acute COPD exacerbation: a pilot study. Respir Res, 2021, 22(1): 30. doi: 10.1186/s12931-020-01610-x.

[24] TAN D, WANG B, CAO P, et al. High flow nasal cannula oxygen therapy versus non-invasive ventilation for acute exacerbations of chronic obstructive pulmonary disease with acute-moderate hypercapnic respiratory failure: a randomized controlled non-inferiority trial. Crit Care, 2024, 28(1): 250. doi: 10.1186/s13054-024-05040-9.

[25] JERATH R, BEVERIDGE C. Respiratory rhythm, autonomic modulation, and the spectrum of emotions: the future of emotion recognition and modulation. Front Psychol, 2020, 11: 1980. doi: 10.3389/fpsyg.2020.01980.

[26] PATEL A, ANUSSHREE R, SWARNA M, et al. Detection of atrial fibrillation using PPG signals. 2024 IEEE 21st India Council International Conference (INDICON), 2024: 1-6. doi: 10.1109/INDICON63790.2024.10958406.

[27] ABRISHAM K P, ALIPOUR K, TARVIRDIZADEH B, et al. Noninvasive assessment of arterial stiffness using photoplethysmography: feature analysis and machine learning-based estimation of carotid-femoral pulse wave velocity. IEEE Access, 2025(13): 185657-185673. doi: 10.1109/ACCESS.2025.3626252.

基金资助

*科技创新2030-“脑科学与类脑研究”重大项目(2021ZD0200500)