Design And Implementation Of Eye-Control Interaction System For Arm-Hand Prosthesis

There is a huge number of physically disabled patient in the world,arm-hand robots are expected to facilitate the daily life of patients with physical disabilities,however,it is difficult for patients with upper limbs disability to control arm-hand robots through traditional human-computer interaction(HRI)modality,To this end,a concept of using human eyes to realize HRI is proposed in this paper,and then an eye-hand coordination prosthetic arm-hand system controlled by human gaze is designed and implemented based on the concept.The main work of this paper includes:(1)Target object recognition.Aiming at the problem of target object extraction,a pupil detection algorithm is proposed to extract the human gaze information,and the gaze information is mapped by Gaussian process regression algorithm to obtain the 2D gaze in the scene image,then the fused target recognition algorithm and tracking algorithm extracts bounding boxes of common objects in the scene image,finally the target object can be identified according to the positional relationship between the 2D gaze and the bounding boxes.(2)Pose estimation and coordinate system unification.For the target object localization problem,the pose of the target object in the left foreground camera coordinate system was estimated by binocular 3D reconstruction algorithm and attitude estimation algorithm,and tracks the eye tracker and prosthetic arm in real time using the lighthouse positioning technology,then the transformation relationship of the coordinate system of each module in the system is established and the pose of target object can be transformed into the base coordinate system of the prosthetic arm,so as to estimate the pose of the target object in the global coordinate system.(3)Inverse kinematics and error compensation of the prosthetic arm-hand robot.Aiming at the control of arm-hand robot,the working space of the prosthetic arm is traversed to establish a spatial reachable database,and proposed an optimization method to minimize the absolute angle deviation(AAD)for selecting the optimal joint angle combination among the multiple sets of inverse kinematics,then the error estimation of the prosthetic arm is implemented based on the lighthouse framework and a method for feedback control and error compensation of the prosthetic arm is proposed.(4)Target object grasping experiment.A set of experiments were designed based on the head-mounted eye tracker and the prosthetic arm-hand platform,through the analysis of the experimental data,the grabbing function of the system based on eye-hand coordination was tested and verified.The experimental result shows that the system runs stably with high precision and the prosthetic arm can effectively grasp the target object of the user through gaze recognition.