On-board Image Based Multi-object Tracking And Motion Prediction For Vulnerable Road Users

In the vehicle driving environment,pedestrians and different sorts of riders belong to Vulnerable Road Users(VRU),who have variable appearance properties and random movement characteristics.Compared with drivers of motor vehicles,they are in limited protection.To better protect VRU,in addition to obtaining the basic location and classification information,it is of vital necessity to track their historical trajectories and predict their future trajectories to achieve comprehensive perception.However,the existing environmental perception methods are lack of research for all kinds of VRU,while the perception performance is also limited in the complex and changeable road environment.Explicitly,most existing object detection methods tend to confuse different types of VRU;the multiple object tracking methods are easy to lose the fast-moving objects;the trajectory prediction methods do not make full use of the environmental information,resulting in insufficient accuracy and stability of final motion prediction.To address the above problems,this paper establishes an integrated perception architecture for VRU-oriented object detection,multiple object tracking,and trajectory prediction.The key technologies involved in the process of VRU protection are profoundly studied,which cover dynamic multi-object detection based on deep neural network,online three-dimensional(3D)multiple object tracking considering ego-motion compensation,and multi-object trajectory prediction based on sequence-to-sequence encoding and decoding network.To realize the joint detection of various VRU objects,a deep neural network model under “pre-process-backbone network-neck network-head network-post-process”framework is established,based on the design idea of residual network and feature pyramid.This model integrates network optimization schemes such as structural uncertainty estimation,VRU multi-level hierarchical classification,multi-scale fusion and multi-branch learning.Hence,it can quickly locate and accurately classify various dynamic VRU objects in complex driving environment,and provide reliable detection results for the follow-up object tracking and trajectory prediction.To improve the capability of multi-object tracking for all VRU objects in complex road environments,a Tracking-by-Detection framework-based 3D multiple object tracking method with online updating ability considering self-vehicle motion compensation is proposed.On the basis of the object detection results,this method learns the online motion model of VRU based on the variant model of recurrent neural network.Concurrently,it integrates multi-cascade data association and multi-state life cycle management technologies,and uses self-vehicle motion compensation to do the3 D position optimization,so as to achieve long-time stable and reliable tracking of multi-type multi-object VRU from on-board perspective.To guarantee the motion prediction performance of VRU in a long-term domain,a multiple object trajectory prediction method based on multi-trajectory prediction factors and sequential encoder-decoder network is proposed on the basis of dynamic multi-object detection and online multi-object tracking.The method combines multi-factor motion and multi-dimensional appearance features,as well as contextual interaction features considering motion intention.And it finally outputs stable and reliable future trajectories by utilizing the well-designed encoder-decoder network to learn the mapping from historical sequences to future sequences.In order to verify the effectiveness of the proposed method,experiments are carried out on the new-established comprehensive VRU perception dataset.The results demonstrate that the proposed method can realize joint positioning and classification of all kinds of VRU objects,as well as stable tracking of historical trajectories and the effective prediction of future trajectories,which provide rich and accurate perceptual information for intelligent vehicles.