Designing Affinity Model For Multiple Object Tracking Based On Deep Learning

Multi-object tracking(MOT)aims to estimate the trajectories of multiple objects in a video or images sequence,which is a highly active research in the field of computer vision and pattern recognition.As a foundation for the following analysis and understanding of video content,MOT has a wide range of applications such as intelligent video surveillance,autonomous driving,human computer interaction,medical diagnosis,precision guidance and many other fields.Although much progress has been reported,multi-object tracking remains very challenging due to the change of background clutters,frequent occlusions and variable number of targets.Tracking-by-detection has become the primary approach in multi-object tracking thanks to the recent breakthrough in deep learning and significant improvements on object detection.In this paradigm,a detector pre-trained offline firstly generates the position of each target in each frame(i.e.detection response),and the multi-object tracking is then formulated as a data association problem,where the detection responses belonging to the same target are associated and the complete target trajectories for each target are obtained.Consequently,how to design an affinity model is crucial in tracking-by-detection approaches.The goal of an affinity model is to calculate the linking probability between detections or tracklets.To performing tracking in complex scenes,an affinity model extracts discriminative feature of the target,and should be robust to the variation between intra-target yet remains discriminative for the inter-target variation.Under the tracking-by-detection framework,the focus of the thesis is to exploit deep learning technique to design affinity model for multi-object tracking,and the main contributions are given below.(1)Design affinity model based on deep metric learningThere exists an important similarity between multi-object tracking and person re-identification.Based on this observation,this paper applies the triplet loss function,which is widely used in person re-identification,to multi-target tracking field for the first time.By collecting triplet samples with triplet constraint,a three-channel convolution neural network is trained to extract appearance features.We calculate the similarity of appearance between targets using discriminative appearance features,and combine it with linear motion model to obtain the associating probability between trajectories.The adaptive time sliding window and the Hungarian algorithm are adopted for multi-level association to obtain the target trajectory.Experiments on the challenging MOT database demonstrate the validity of the algorithm.(2)Design affinity model based on feature fusion and metric learning jointlyDue to the complexity of the tracking scene and uncertainty of the target motion pattern,e.g.,abrupt change of moving direction and speed,severe jitter of the camera.In these cases,a simple linear motion model cannot extract accurate target motion feature.To tackle this problem,a recurrent neural network is employed to model the nonlinear motion of targets for extracting motion features.By combining appearance and motion cues,we propose an affinity model based on feature fusion and deep metric learning which could be learned in an end-to-end manner.The proposed method achieves comparable or superior tracking results when compared with the state-of-the-art approaches.(3)Multi-object tracking based on recurrent neural networks and Bayesian filteringMost existing tracking-by-detection algorithms focus on how to realize reliable data association.On the other hand,detection responses outputted by a detector have influenced the performance of MOT algorithms.To overcome detector's limitation such as missed detections,false alarms and inaccurate localization,a trajectory estimation strategy is presented.Specifically,a recurrent neural networks(RNN)that is embedded in the above motion model is designed.The RNN takes hidden state of the above LSTM network as input and conducts recursive prediction and update for explicitly estimating targets state.Experimental results show that the proposed trajectory estimation strategy can further improve MOT performance and achieve more accurate tracking localization.