Activity based geometry independent features for information processing in heterogeneous camera networks

Heterogeneous surveillance camera networks permit pervasive, wide-area visual surveillance for urban environments. However, due to the vast amounts of data they produce, human-operator monitoring is not possible and automatic algorithms are needed. In order to develop these automatic algorithms efficient and effective multi-camera information processing techniques must be developed. However, such multi-camera information processing techniques pose significant challenges in heterogeneous networks due to the fact that (i) most intuitive features used in video processing are geometric, i.e. utilize spatial information present in the video frames, (ii) the camera topology in heterogeneous networks is dynamic and cameras have significantly different observation geometries, consequently geometric features are not amenable to devising simple and efficient information processing techniques for heterogeneous networks.;Based on these observations, we propose activity based behavior features that have certain geometry independence properties. Specifically, when the proposed features are used for information processing applications, a location observed by a number of cameras generates the same features across the cameras irrespective of their locations, orientations, and zoom levels. This geometry invariance property significantly simplifies the multi-camera information processing task in the sense that network's topology and camera calibration are no longer necessary to fuse information across cameras. We present applications of the proposed features to two such problems: (i) multi-camera correspondence, (ii) multi-camera anomaly detection.;In the multi-camera correspondence application we use the activity features and propose a correspondence method that is robust to pose, illumination & geometric effects, and unsupervised (does not require any calibration objects to be utilized). In addition, through exploitation of sparsity of activity features combined with compressed sensing principles, we demonstrate that the proposed method is amenable to low communication bandwidth which is important for distributed systems. We present quantitative and qualitative results with synthetic and real life examples, which demonstrate that the proposed correspondence method outperforms methods that utilize geometric features when the cameras observe a scene with significantly different orientations.;In the second application we consider the problem of abnormal behavior detection in heterogeneous networks, i.e., identification of objects whose behavior differs from behavior typically observed. We develop a framework that learns the behavior model at various regions of the video frames, and performs abnormal behavior detection via statistical methods. We show that due to the geometry independence property of the proposed features, models of normal activity obtained in one camera can be used as surrogate models in another camera to successfully perform anomaly detection. We present performance curves to demonstrate that in realistic urban monitoring scenarios, model training times can be significantly reduced when a new camera is added to a network of cameras. In both of these applications the main enabling principle is the geometry independence of the chosen features, which demonstrates how complex multi-camera information processing problems can be simplified by exploiting this principle.;Finally, we present some statistical developments in the wider area of anomaly detection, which is motivated by the abnormal behavior detection application. We propose test statistics for detection problems with multidimensional observations and present optimality and robustness results.