Research On Remote Robot Demonstration System Integrated With Mixed Reality Environment

In recent years,robots have gradually become one of the hot topics in technology,economy,and society,and are also gradually being used in various industries,such as industrial production,home services,education,and entertainment.With the increasing demand for human-machine interaction,for the purpose of the task and the safety of human-robot interaction,the perception function of the robot is becoming more and more important.This article mainly explores the problem of combining virtual reality scenes with robot teaching.Robot programming by demonstration is an important research branch in the current robot application field.Users teach skills to robots through demonstration,so that the skills learned by robots can be highly humanoid.Skill transfer and sharing are effective ways for robots to efficiently acquire operating skills.Traditional industrial robots usually use hand-held programmers to implement motion planning,which is time-consuming,labor-intensive,and result in rigid movement.Besides,simulator-based off-line programming software is difficult to use when the environment and object models are inaccurate.How to efficiently complete an assembly task is in the hands of assembly line workers with long working experience,it is an important research topic for human-machine integration to let skilled workers teach robots hand in hand based on their own operating characteristics and realize the transfer of skills from humans to robots.In the traditional demonstration process,the operator always needs to have close interaction with the robot.in this way,the operator's personal safety is threatened in a dangerous environment or when the robot fails.Virtual / Mixed Reality(VR / MR)is a popular technology that can project virtual scenes into the real environment.Combining this technology with robot demonstration can easily achieve the needs of teleoperation and ensure operator's safety under the premise of providing them with a telepresence experience.Aiming at long-distance demonstrations,this paper proposes a cross-platform industrial robot demonstration system based on mixed reality technology,which applies gesture recognition to extract the operator's hand motion signals,then realizes the remote real-time drag operation of the robot manipulator.By designing a mixed reality environment,the virtual model of the operator's hand is combined with the actual working environment of the robot,and the headmounted display is used to display the operation guidance,which further improves the presence of the operation process and effectively improves demonstrate accuracy to provide operators with efficient and realistic demonstrating experience.In addition,this paper also proposes a multi-sensor data fusion method to address the loss tracking problem of the gesture controller Leap Motion to improve the robustness of the gesture controller and achieve full-position palm position detection.Finally,this paper designs a kind of telemanipulation demonstration platform for mechanical dexterous hands.By applying gesture sensors,the operator's hand movement data is acquired,which is then used to control a five-finger humanoid manipulator and restore the operator's hand dynamics in real-time to achieve grabbing of irregularly shaped objects.