基于内窥镜视觉的机器人辅助手术中力估计方法

Force Estimation Method in Robot-Assisted Surgery Based on Endoscopic Vision

摘要

在机器人辅助微创手术中，精确的力反馈对于提高手术操作的安全性与质量至关重要．然而，现有的解决方案在实际应用场景中仍面临着如小型化、精准化和普适性等多重挑战，限制了其在复杂临床场景中的广泛应用．因此，如何实现高精度、低成本且适用于多种组织类型的力估计成为研究重点．为此，基于手术机器人配备的内窥镜系统，构建了包含多种材料和丰富力学信息数据集，并提出了结合注意力机制的深度学习模型，以优化内窥镜视觉信息的特征提取，从而提升力估计的准确性和鲁棒性．模型以材料变形的内窥镜图像作为输入，结合卷积神经网络的特征提取能力和循环神经网络的时序建模能力，实现对手术器械与组织之间交互力的精确估计．此外，为进一步探究注意力机制在力估计任务中的作用和优化策略，提出了3种不同的注意力模块引入方案．实验结果表明：采用DenseNet-BiLSTM结构并引入SENet模块的模型在3种材料上取得了最佳性能，显著提升了模型的整体表现；同时，注意力模块的位置对不同组织材料的力估计效果具有差异性影响．研究验证了基于内窥镜视觉的深度学习方法在精确估计手术过程中器械与组织间的交互力方面的有效性和可行性，为未来机器人辅助微创手术系统的发展和优化提供了新的方向与理论依据．

Abstract

Precise force feedback is crucial for improving the safety and quality of operations in robot-assisted minimally invasive surgery(RAMIS). However, existing solutions face challenges related to miniaturization, precision, and adaptability, limiting their application in complex clinical scenarios. Addressing these limitations has made the development of high-accuracy, low-cost force estimation methods for various tissue types a key research focus. To address this, a dataset was created using an endoscope integrated with a surgical robot. This dataset includes various material types and rich mechanical interaction data. Deep learning models, incorporating attention mechanisms, were developed to optimize visual feature extraction from endoscopic images and improve the accuracy and robustness of force estimation. These models used endoscopic images of material deformation as inputs, combining the feature extraction strengths of convolutional neural networks with the temporal modeling capabilities of recurrent neural networks to estimate the interaction forces between surgical instruments and tissues accurately. Three integration schemes for attention modules were proposed to further explore their roles and optimization strategies in force estimation. Experimental results show that the DenseNet-BiLSTM architecture, when combined with the SENet module, derives the best performance across three material types, significantly improving the model’s overall performance. The attention module’s placement has varying effects on force estimation accuracy, depending on the material type. These findings validate the effectiveness and feasibility of using endoscopic vision-based deep learning methods for accurately estimating interaction force during surgery. They pave the way for new directions and provide theoretical foundations for the development and optimization of future RAMIS systems.

关键词

机器人辅助微创手术 / 视觉反馈 / 交互力估计 / 注意力机制 / 深度学习

Key words

robot-assisted minimally invasive surgery(RAMIS) / visual feedback / interaction force estimation / attention mechanism / deep learning

参考文献

[1] Gomes P. Surgical robotics:Reviewing the past, analysing the present, imagining the future[J]. Robotics and Computer—Integrated Manufacturing, 2011, 27(2):261-266.

[2] Zareinia K, Maddahi Y, Gan L S, et al. A force—sensing bipolar forceps to quantify tool—tissue interaction forces in microsurgery[J]. IEEE/ASME Transactions on Mechatronics, 2016, 21(5):2365-2377.

[3] Demi B, Ortmaier T, Seibold U. The touch and feel in minimally invasive surgery[C]// IEEE International Workshop on Haptic Audio Visual Environments and Their Applications. Ottawa, Canada, 2005:1-6.

[4] Wagner C R, Stylopoulos N, Jackson P G, et al. The benefit of force feedback in surgery:Examination of blunt dissection[J]. Presence:Teleoperators and Virtual Environments, 2007, 16(3):252-262.

[5] Reiley C E, Akinbiyi T, Burschka D, et al. Effects of visual force feedback on robot—assisted surgical task performance[J]. The Journal of Thoracic and Cardiovascular Surgery, 2008, 135(1):196-202.

[6] Okamura A M. Haptic feedback in robot—assisted minimally invasive surgery[J]. Current Opinion in Urology, 2009, 19(1):102-107.

[7] Marban A, Srinivasan V, Samek W, et al. A recurrent convolutional neural network approach for sensorless force estimation in robotic surgery[J]. Biomedical Signal Processing and Control, 2019, 50:134-150.

[8] Zhao B, Nelson C A. A sensorless force—feedback system for robot—assisted laparoscopic surgery[J]. Computer Assisted Surgery, 2019, 24(Suppl1):36-43.

[9] Xue R, Du Z, Yan Z, et al. An estimation method of grasping force for laparoscope surgical robot based on the model of a cable—pulley system[J]. Mechanism and Machine Theory, 2019, 134:440-454.

[10] Jung W J, Kwak K S, Lim S C. Vision—based suture tensile force estimation in robotic surgery[J]. Sensors, 2021, 21(1):110.

[11] Naeini F B, Makris D, Gan D, et al. Dynamic—vision—based force measurements using convolutional recurrent neural networks[J]. Sensors, 2020, 20(16):4469.

[12] Arm R, Shahidi A, Clarke C, et al. Synthesis and characterisation of a cancerous liver for presurgical planning and training applications[J]. BMJ Open Gastroenterology, 2022, 9(1):e000909.

[13] Horeman T, Rodrigues S P, Jansen F W, et al. Force measurement platform for training and assessment of laparoscopic skills[J]. Surgical Endoscopy, 2010, 24:3102-3108.

[14] Sabique P V, Pasupathy G, Ramachandran S, et al. Investigating the influence of dimensionality reduction on force estimation in robotic—assisted surgery using recurrent and convolutional networks[J]. Engineering Applications of Artificial Intelligence, 2023, 126:107045.

[15] Huang G, Liu Z, van der Maaten L, et al. Densely connected convolutional networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA, 2017:4700-4708.

[16] Hu J, Shen L, Sun G. Squeeze—and—excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018:7132-7141.

[17] Wang Q, Wu B, Zhu P, et al. ECA—Net:Efficient channel attention for deep convolutional neural networks[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA, 2020:11534-11542.

[18] Girshick R. Fast R—CNN[C]// Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile, 2015:1440-1448.

[19] Smith L N. Cyclical learning rates for training neural networks[C]// 2017 IEEE Winter Conference on Applications of Computer Vision(WACV). Santa Rosa, USA, 2017:464-472.

[20] Seber G A F, Lee A J. Linear Regression Analysis[M]. Hoboken: John Wiley & Sons, 2012.

[21] 洪鹰, 徐世超, 肖聚亮, 等. 基于在线辨识的机器人惯量前馈控制仿真研究[J]. 天津大学学报(自然科学与工程技术版), 2019, 52(10):1069-1080.

[22] Hong Ying, Xu Shichao, Xiao Juliang, et al. Simulation study of robot inertia feedforward control based on online identification[J]. Journal of Tianjin University(Science and Technology), 2019, 52(10):1069-1080(in Chinese).

基金资助

国家重点研发计划资助项目(2022YFC2409603)