Human re-identification in real-world surveillance camera networks

Video surveillance has become critical for security applications. With cameras and data storage devices getting more affordable, many institutions and organizations have chosen to install camera networks for safety and surveillance. The U.S. Department of Homeland Security, for instance, has directed a huge amount of manpower and expenditure to the installation, maintenance, replacement and operation of surveillance camera networks. Public transportation centers such as airports, train stations, and bus stops are some of the most concentrated environments. The traditional monitor- ing method, which completely relies on security officers' observations, becomes less feasible when more and more screens need to be watched at the same time. Instead, video analytic solutions that process multiple cameras simultaneously are more reliable. In this thesis, we focus on one particular application, human re-identification, with a focus on the challenges of real-world scenarios.;First, we propose a viewpoint invariant human re-identification framework that considers pose information and person-specific discriminative features. We observe that appearance consistency at different spacial locations varies with the object's pose. A pixel-wise weighting scheme is learned to provide the local robustness to varying viewpoint. For the appearance model matching, instead of a universal distance metric, we further learn a discriminative weighting scheme for a given person. Experimental results show that the proposed method boosts the performance of various state-of- the-art metric learning approaches.;Next, we introduce an efficient metric learning algorithm for the multi-shot re- identification problem based on a combination of random projections and random forests. To obtain a high matching rate, re-identification algorithms usually employ feature vectors with high dimensionality. This severely affects the computational efficiency in metric learning and matching. We use random projections to transfer all the calculations into a very small subspace and boost the performance by accumulating the metrics learned from random forests in each subspace. With the randomness brought by both techniques, we substantially increase the diversity of training data samples and produce a robust ensemble model.;Third, we propose a multi-shot human re-identification framework to address the issue of effectively using image sequences from tracking results. Because of the multi-modal nature of the feature data distribution with respect to different cameras, Local Fisher Discriminant Analysis (LFDA) is particularly suitable to provide a sub- space where feature data from different people are maximally separated while the local structure of each class is still preserved. A clustering step is adopted to further elim- inate difficulties introduced by the multimodality of the feature data from the same image sequence, so that LFDA is able to better separate the classes. The relationship between different cameras is established by a metric learning step. This algorithm models the appearance characteristic of each person directly from the tracking results. It is very efficient because an analytic solution is achievable for the dimensionality reduction step and metric learning is performed in a lower dimensional space.;Finally, we introduce an end-to-end human re-identification solution installed in a mid-size U.S airport. Designing and building a fully functional human re- identification software embedded into a real-world surveillance system involves many challenges that are not encountered in typical re-identification research. For instance, one needs to implement essential supporting modules such as video streaming, hu- man detection and tracking, as well as a user friendly interface. Moreover, it is critical for the system to run in real-time so that it can handle live re-identification requests. Real-world issues such as complicated illumination conditions, low resolution images and crowded scenes should also be taken into consideration. We describe the high- level system design and the algorithm framework of our re-identification solution. We discuss above challenges and trade-offs, as well as initial results that show the perfor- mance of the algorithms in the real-world surveillance task.