Research On Head Pose Estimation And Follow-up Control Based On Multi-sensor

With the development of computer vision and AR technology,people have more and more demands for stereo vision applications,such as assisted driving for special vehicles and assisted control for unmanned engineering vehicles.The occupants of special vehicles directly observe the situation outside the vehicle through the observation port of the vehicle,which has a relatively large field of vision.A visual system that can conform to the human visual habits is required to improve the observation range and convenience.Unmanned engineering vehicles（such as excavators）operating in hazardous environments also require a vision system that can realize remote and efficient remote control operations.Therefore,this thesis proposes and implements a set of head posture follow-up control vision system,which can provide passengers with realistic stereo vision,infrared video suitable for night driving and variable zoom video for quickly locking observation targets.The orientation of the high-speed follow-up PTZ equipped with the camera is always consistent with the orientation of the human head,which is compatible with the general visual habits of human,making the system have low learning cost and use experience;In addition,head control pan tilt can liberate people’s hands to do other operations,which can greatly improve the work efficiency of personnel.The traditional acquisition of human head posture is mainly divided into two methods: image posture estimation and inertial sensor posture calculation.Image pose estimation methods can be divided into two types of head pose estimation methods without fiducial markers and with fiducial markers.The results obtained by the head pose estimation method without fiducial markers generally have poor accuracy and a large amount of computation;The accuracy of the head pose estimation method with fiducial markers has been greatly improved,but the performance of the obtained pose data is limited by the frame rate of the camera,and the detectable angle range of the plane fiducial marker target is limited.The inertial sensor attitude calculation method is mainly divided into two types of inertial attitude measurement units: 6-axis IMU and 9-axis IMU.The 6-axis IMU cannot operate reliably for a long time due to the cumulative error of the horizontal attitude angle.The 9-axis IMU corrected by the magnetometer solves the problem of cumulative error,but it is sensitive to changes in the surrounding magnetic field,so it is not suitable for the application scenarios of special vehicles and engineering vehicles with complex magnetic field conditions in this thesis.This thesis proposes a multi-sensor human head pose estimation method that integrates camera,gyroscope,and accelerometer data,designs and implements a stereo detection target with multiple fiducial markers,and fixes the fiducial marker codes on multiple planes of the detection target,so that the attitude of each plane marking code can be detected separately,and then the coordinate system of each plane can be converted into a unified coordinate system by using the transformation relationship between the planes,that is,the attitude angle data obtained by each plane can be converted into a unified world.In the coordinate system,this greatly improves the detectable range of the camera attitude estimation,and the attitude data of the detection target can be obtained by processing the pictures of the detection target captured by the camera.A 6-axis head attitude acquisition module and a 9-axis head attitude acquisition module are designed and implemented,including the chip selection of the two modules,the design and implementation of the hardware circuit,and the implementation of the software in the modules.Finally,the extended Kalman filter algorithm is used to fuse the camera image pose estimation result with the head pose acquisition module to realize the multi-sensor head pose estimation.By comparing and analyzing the attitude data obtained by the camera attitude estimation method of the fiducial marker and the inertial sensor method,the multi-sensor attitude estimation method of the fusion camera,gyroscope and accelerometer proposed in this thesis solves the problem of accumulated error and reduces the high frequency error.At the same time,the influence of magnetic interference is completely eliminated.Through the experimental test,the attitude data error of the attitude estimation method proposed in this thesis is within ±1°,and the error of the system follow-up control is within ±2°.In addition,experimental tests show that other functions of the system such as stereo vision and video display of the host computer in the head-mounted display are also realized normally.Finally,the system designed in this thesis is applied to the vehicle for outdoor experimental tests.The test results show that the system works well and meets the design requirements.