To address the problems of insufficient global semantic modeling, limited local geometric representation, and difficulty in unified representation of multi-scale features in complex scenes in grasp pose estimation, this paper conducted research centered on multi-level feature collaborative modeling and constructed corresponding network methods on two perception representations: two-dimensional images and three-dimensional point clouds. For two-dimensional planar grasp tasks, PTUNet with parallel Transformer and ResUnet++ was proposed. Through the bidirectional interaction between global semantic branch and local detail branch, the joint prediction of grasp quality, angle, and width was realized. For six-degree-of-freedom grasp tasks, CascadeGraspNet was proposed. Through the cascading local-global optimization module and multi-scale cylinder sampling module, the abilities of local structure perception, global context modeling, and cross-scale geometric aggregation of point clouds were enhanced. Experimental results indicate that PTUNet grasp accuracy reaches 97.8% on the Jacquard dataset; CascadeGraspNet average precision for known objects, similar objects, and novel objects reaches 71.39%, 63.29%, and 28.69%, respectively, on the RealSense subset of GraspNet-1Billion dataset, and it achieves the grasp success rate of 97.8% in real-scene grasp experiments. Therefore, multi-level feature collaborative modeling can effectively improve the accuracy, robustness, and generalization capability of grasp pose estimation.
现有方法可分为面向二维图像的平面抓取估计和面向点云的六自由度抓取估计两类.前者通常直接从RGB(red green blue)或RGB-D(red green blue-depth)图像中预测抓取位置、角度和夹爪宽度[2-4],具有计算效率高、部署成本低等优点,但传统卷积结构受限于局部感受野,难以稳定刻画目标整体形状、主轴方向及其与背景之间的长距离依赖关系;后者能够在三维空间中表达更灵活的抓取位姿,更适合处理遮挡、堆叠和非结构化场景[5-8],但局部几何细节提取、全局上下文建模以及多尺度邻域建模之间往往难以兼顾.
为了避免双分支仅形成并行堆叠而缺乏有效协同,PTUNet设计了特征交互模块(feature interaction module,FIM),在编码与解码阶段分别引入双向特征交互模块.图2展示了RtoT(ResUNet to Transformer)模块的结构,该模块在编码阶段将RU分支提取的高层局部语义特征映射到Transformer嵌入空间,使ST分支在全局建模过程中能够保留更多边缘和结构细节信息.
图3展示了TtoR(Transformer to ResUNet)模块的结构,该模块在解码阶段将ST分支中经全局建模后的上下文特征回注到RU分支,从而增强局部特征与整体语义之间的一致性.
BohgJ, MoralesA, AsfourT, et al. Data-driven grasp synthesis: a survey[J]. IEEE Transactions on Robotics, 2014, 30(2): 289-309.
[2]
JiangY, MosesonS, SaxenaA. Efficient grasping from RGBD images: learning using a new rectangle representation[C]//2011 IEEE International Conference on Robotics and Automation. Piscataway, NJ: IEEE, 2011: 3304-3311.
[3]
LenzI, LeeH, SaxenaA. Deep learning for detecting robotic grasps[J]. The International Journal of Robotics Research, 2015, 34(4/5): 705-724.
[4]
RedmonJ, AngelovaA. Real-time grasp detection using convolutional neural networks[C]//2015 IEEE International Conference on Robotics and Automation. Seattle: IEEE, 2015: 1316-1322.
[5]
ten PasA, GualtieriM, SaenkoK, et al. Grasp pose detection in point clouds[J]. The International Journal of Robotics Research, 2017, 36(13/14): 1455-1473.
[6]
MuraliA, MousavianA, EppnerC, et al. 6-DOF grasping for target-driven object manipulation in clutter[C]//2020 IEEE International Conference on Robotics and Automation. Paris: IEEE, 2020: 6232-6238.
[7]
QinY Z, ChenR, ZhuH, et al. S4G: amodal single-view single-shot SE(3) grasp detection in cluttered scenes[C]//Proceedings of the 4th Conference on Robot Learning. London: PMLR, 2020: 53-65.
[8]
LiangH Z, MaX J, LiS, et al. PointNetGPD: detecting grasp configurations from point sets[C]//2019 International Conference on Robotics and Automation. Montreal: IEEE, 2019: 3629-3635.
[9]
VaswaniA, ShazeerN, ParmarN, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017: 6000-6010.
[10]
LiuZ, LinY T, CaoY, et al. Swin Transformer: hierarchical vision Transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 10012-10022.
[11]
JhaD, SmedsrudP H, RieglerM A, et al. ResUnet++: an advanced architecture for medical image segmentation[C]//2019 IEEE International Symposium on Multimedia. San Diego: IEEE, 2019: 225-230.
[12]
GokcesuK, GokcesuH. Generalized Huber loss for robust learning and its efficient minimization for a robust statistics[EB/OL]. (2021-08-21)[2026-02-26].
[13]
ChoyC, GwakJ, SavareseS. 4D spatio-temporal ConvNets: Minkowski convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 3075-3084.
[14]
DepierreA, DellandréaE, ChenL M. Jacquard: a large scale dataset for robotic grasp detection[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid: IEEE, 2018: 3511-3516.
[15]
ChangA X, FunkhouserT, GuibasL, et al. ShapeNet: an information-rich 3D model repository[EB/OL]. (2015-12-10)[2026-02-26].
[16]
FangH S, WangC X, GouM H, et al. GraspNet-1Billion: a large-scale benchmark for general object grasping[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11444-11453.
[17]
MorrisonD, CorkeP, LeitnerJ. Learning robust, real-time, reactive robotic grasping[J]. The International Journal of Robotics Research, 2020, 39(2/3): 183-201.
[18]
ZhouX W, LanX G, ZhangH B, et al. Fully convolutional grasp detection network with oriented anchor box[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid: IEEE, 2018: 7223-7230.
[19]
GariépyA, RuelJ C, Chaib-draaB, et al. GQ-STN: optimizing one-shot grasp detection based on robustness classifier[C]//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Macau: IEEE, 2019: 3996-4003.
[20]
ZhangH B, LanX G, BaiS T, et al. ROI-based robotic grasp detection for object overlapping scenes[C]//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Macau: IEEE, 2019: 4768-4775.
[21]
AinetterS, FraundorferF. End-to-end trainable deep neural network for robotic grasp detection and semantic segmentation from RGB[C]//2021 IEEE International Conference on Robotics and Automation. Xi’an: IEEE, 2021: 13452-13458.
[22]
KumraS, KananC. Robotic grasp detection using deep convolutional neural networks[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems. Vancouver: IEEE, 2017: 769-776.
[23]
WangS C, ZhouZ L, KanZ. When Transformer meets robotic grasping: exploits context for efficient grasp detection[J]. IEEE Robotics and Automation Letters, 2022, 7(3): 8170-8177.