Research On Distributed Multi-view Information Fusion For 3D Human Pose Estimation

Human Pose Estimation is an important research direction in computer vision and has a wide application prospect in Smart Home,Intelligent Surveillance,Sports Medicine,Human-Computer Interaction,Virtual Reality and other fields.However,as a result of the self-occlusion of human body action,the video shot from a single view can not visualize the accurate action.Therefore,it is of great scientific significance and application value to study the fusion of visual information from multi-view,to obtain more accurate three-dimensional human body posture and to solve the problem of self-occlusion.At present,Human Behavior Perception based on sensor network is an effective way to solve self-occlusion from single-view vision,especially the RGBD 3D camera Kinect launched by Microsoft,which can directly obtain the underlying scene information and extract 3D coordinates of human joints,thus is conducive to realize information fusion from multi-view.However,the current Multi-vision Human Pose Estimation based on RGBD 3D camera network still has many challenges as follows:1)the real-time transmission of massive raw data between network nodes is challenging due to the limitation of network bandwidth;2)the lack of camera network data sets with calibration brings difficulty to evaluation of Multi-angle Information Fusion Algorithm;3)limited by the robustness of joint points extraction algorithm,the location of joint points extraction varies greatly from different views,which further obstructs the multi-angle joint data fusion.Addressing these issues,this study constructs a distributed camera network,reduces network load by using human joint point information fusion instead of multi-view 3d point cloud reconstruction,and collects multi-angle human behavior data sets with calibration to provide support for joint point estimation and multi-angle information fusion.In the camera network,distributed fusion algorithm and interactive multi-model make the fused joint points more accurate and easy to identify.The detailed contents and innovations of this study include:1.We construct a small distributed depth camera network with the use of four Kinect V2 and Nvidia Jetson TX2 embedded AI computing equipment.Based on the method of human joint point estimation and distributed filtering algorithm,with the use of camera network,a distributed human joint point estimation system is constructed in Robot Operating System environment.The system is more scalable and fault-tolerant than the current centralized human joint point estimation system.Compared with the current commercial optical human motion capture system,its low cost,easy operation and no need to wear additional markers are very suitable for the behavior recognition direction of people in daily living environment.Addressing the problem of a lack of multi-angle human behavior data set with calibration,the DATASET SET?DATASET OP data sets were collected in this network environment,imitating MSRAction3D.These two data sets contain many human behaviors,including daily activities,which can provide strong support for human joint point extraction,multi-angle information fusion,and lay the foundation for online behavior recognition.2.For Kinect SDK extraction of human joint points can not identify the positive and negative aspects of human body,we propose a method to estimate 3D joint points combined with Multi-point Co-circle Method,based on 2D joint points and depth maps,which can solve the former problem with the use of two-dimensional pose estimation and depth information.According to the idea of Vicon motion capture system to obtain the joint points of human body,this method optimizes the three-dimensional human joint with the use of Multi-point Co-circle Method.To solve the problem that 2D joint points confidence acquired by Openpose can not be applied in 3D joint points and the 3D joint points confidence provided by Kinect SDK is not accurate,we propose to measure the overall joint points confidence by the azimuth of the body plane and to use the occlusion relation between the joints to further optimize local joint points confidence.This method is used on two self-built data sets,which can effectively reduce the influence of wrong joint points confidence on data fusion and facilitate the acquisition of more accurate human pose after data fusion.3.Aiming at the problem of single angle occlusion,self-occlusion and the motion of human joints is not a single uniform motion model,a multi-angle information fusion method is proposed,which is consist of distributed information consistency algorithm and interactive multi-model.When body points are moving,the joint point motion of human body can be fitted by switching between some motion models,such as uniform speed and uniform acceleration,which can solve the problem that the model of average velocity motion can not describe the randomness and mobility of human motion.With the use of distributed fusion algorithm,multi-angle data are fused,and the occlusion problem is solved.In addition,distributed fusion algorithm can solve the problem of low fault tolerance and weak expansion of centralized fusion algorithm.The combination of interactive multi-model and distributed fusion algorithm can provide more accurate and easy to recognize human joint points for human behavior recognition.This study construct a human joint point estimation system,which combines camera network,human pose estimation and distributed information fusion closely.It is low cost,expandable,and has unlabeled performance and high fault tolerance,can provide accurate and easily-to-identify 3D human joint points.Thus the distributed system has good practical value and scientific significance for solving single-view occlusion,improving multi-view information fusion effect and human behavior recognition rate.