基于虚实迁移强化学习的机器人按钮操作策略研究

Research on Robot Button Operation Policy Based on Sim-to-Real Reinforcement Learning

摘要

具身智能概念的快速发展对智能体与物理世界的交互能力提出了更高要求．在以机器人为代表的智能载体与环境的交互过程中,主要依靠力反馈信号以决定其动作输出的任务称为力交互任务,例如零件装配、按钮操作和门窗开合等．针对此类任务交互对象种类繁多、力反馈特性各不相同的挑战,提出了一种具身智能训练方法,基于虚实迁移(sim-to-real)的概念和强化学习方法搭建了机器人高级力交互操作技能学习训练框架,赋予了机器人安全、准确、适应性强大的力交互操作能力．以经典的机器人力交互场景——按钮操作任务为例:首先,基于域随机化的方法在虚拟环境中构建了大量按钮模型,并从接触刚度的角度划分了机器人与按钮之间的接触阶段；然后,模仿人类在按钮操作过程中的感知方式,结合在线刚度估计算法,在虚拟环境下使用近端策略优化(PPO)算法训练机器人的按钮操作技能；最后,通过 sim-to-real 方法将得到的预训练策略直接部署在真实机器人上,在策略迁移后的实机操作实验中得到了良好的结果．在针对具有不同力反馈特性的按钮进行操作的泛化能力测试实验中,经上述方法训练得到的策略展现了远优于现有方法的泛化性能．

Abstract

The rapid development of the concept of embodied intelligence has increased the requirements for the interaction capabilities of agents with the physical world. During the interaction between intelligent carriers,such as robots and the environment,tasks that rely primarily on force feedback signals to determine action outputs are known as force interaction tasks,including component assembly,button operation,and door/window manipulation. To address the challenges posed by the diversity of interaction objects and the varying force feedback characteristics during such tasks,a training method for embodied intelligence was proposed. A training framework for learning advanced robotic force interaction skills was developed based on the sim-to-real concept,as well as reinforcement learning,thereby enabling robots to safely,accurately,and adaptively perform force interaction tasks. Considering the classic robotic force interaction scenario—button operation—as an example,numerous button models were constructed in a virtual environment using domain randomization,and the contact phases between the robot and the button were categorized based on contact stiffness. Thereafter,inspired by human perception of button operation,an online stiffness estimation algorithm was incorporated,and the proximal policy optimization (PPO)algorithm was employed to train the button operation skills of the robot in the virtual environment. Finally,the pretrained policy was directly deployed onto a real robot through the sim-to-real method,achieving favorable results in real-world experiments. In generalization tests on buttons with different force feedback characteristics,the policy,which was trained using the proposed method,demonstrated significantly superior generalization performance compared with the existing approaches.

关键词

机器人 / 技能学习 / 力交互 / 按钮操作 / 强化学习 / 虚实迁移

Key words

robot / skill learning / force interaction / button operation / reinforcement learning / sim-to-real

参考文献

[1] Cheng M, Xiang D. The design and application of a track—type autonomous inspection robot for electrical distribution room[J]. Robotica, 2020, 38(2):185-206.

[2] Sukhoy V, Sinapov J, Wu L, et al. Learning to press doorbell buttons[C]// 2010 IEEE International Conference on Development and Learning. Ann Arbor,USA, 2010:132-139.

[3] Wang F, Chen G, Hauser K. Robot button pressing in human environments[C]// 2018 IEEE International Conference on Robotics and Automation(ICRA). Brisbane,Australia, 2018:7173-7180.

[4] Ravichandar H, Polydoros A S, Chernova S, et al. Recent advances in robot learning from demonstration[J]. Annual Review of Control,Robotics,and Autonomous Systems, 2020, 3(1):297-330.

[5] Brunke L, Greeff M, Hall A W, et al. Safe learning in robotics:From learning—based control to safe reinforcement learning[J]. Annual Review of Control,Robotics,and Autonomous Systems, 2022, 5(1):411-444.

[6] Ren L, Dong J, Liu S, et al. Embodied intelligence toward future smart manufacturing in the era of AI foundation model[J]. IEEE/ASME Transactions on Mechatronics, 2024, 30(4):2632-2642.

[7] Racca M, Pajarinen J, Montebelli A, et al. Learning in—contact control strategies from demonstration[C]// 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Daejeon,Republic of Korea, 2016:688-695.

[8] Liu X, Huang P, Liu Z. A novel contact state estimation method for robot manipulation skill learning via environment dynamics and constraints modeling[J]. IEEE Transactions on Automation Science and Engineering, 2022, 19(4):3903-3913.

[9] Levine S, Pastor P, Krizhevsky A, et al. Learning hand—eye coordination for robotic grasping with deep learning and large—scale data collection[J]. The International Journal of Robotics Research, 2018, 37(4/5):421-436.

[10] Andrychowicz M, Baker B, Chociej M, et al. Learning dexterous in—hand manipulation[J]. The International Journal of Robotics Research, 2020, 39(1):3-20.

[11] Matas J, James S, Davison A J. Sim—to—real reinforcement learning for deformable object manipulation[C]// Conference on Robot Learning. Zurich,Switzerland, 2018:734-743.

[12] Hebecker M, Lambrecht J, Schmitz M. Towards real—world force—sensitive robotic assembly through deep reinforcement learning in simulations[C]// 2021 IEEE/ ASME International Conference on Advanced Intelligent Mechatronics(AIM). Delft,Netherlands, 2021:1045-1051.

[13] Liao Y C, Kim S, Lee B, et al. Button simulation and design via FDVV models[C]// Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Honolulu,USA, 2020:1-14.

[14] Kim S, Lee G. Haptic feedback design for a virtual button along force—displacement curves[C]// Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. St Andrews,UK, 2013:91-96.

[15] Tashiro K, Shiokawa Y, Aono T, et al. Realization of button click feeling by use of ultrasonic vibration and force feedback[C]// World Haptics 2009—3rd Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. Salt Lake City,USA, 2009:1-6.

[16] Elguea—Aguinaco Í, Serrano—Muñoz A, Chrysostomou D, et al. A review on reinforcement learning for contact—rich robotic manipulation tasks[J]. Robotics and Computer—Integrated Manufacturing, 2023, 81:102517.

[17] Jasim I F, Plapper P W. Contact—state monitoring of force—guided robotic assembly tasks using expectation maximization—based Gaussian mixtures models[J]. The International Journal of Advanced Manufacturing Technology, 2014, 73(5/6/7/8):623-633.

[18] Roveda L, Vicentini F, Tosatti L M. Deformation—tracking impedance control in interaction with uncertain environments[C]// 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo,Japan, 2013:1992-1997.

[19] Oulasvirta A, Kim S, Lee B. Neuromechanics of a button press[C]// Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Montreal,Canada, 2018:1-13.

[20] Valverde N, Ribeiro A M R, Henriques E, et al. An engineering perspective on the quality of the automotive push—buttons’ haptic feedback in optimal and suboptimal interactions[J]. Journal of Engineering Design, 2019, 30(8/9):336-367.

基金资助

国家自然科学基金资助项目(52175025)

国家自然科学基金资助项目(52325501)