Research On The Spatial And Temporal Characteristics Of Visual Saliency And Its Applications

With the rapid development of the information processing techniques, people would like to obtain useful information from large amounts of image and video data quickly. Recently, researchers pay more attention to the attention mechanisms. Attention, a cognitive ability, helps us concentrate on the important parts in a complex visual environment and ignore the others. Hence, visual saliency could increase the efficiency of processing and improve the accuracy in the image or video processing. Based on this, we build a tracking algorithm based on the spatio-temporal motion saliency. According to the hierarchical motion processing in visual cortex, we implement a computation model of the spatio-temporal motion saliency. We track the object in the framework of particle filter by combining spatio-temporal motion saliency with the color histogram. In addition, as the saliency in crowd is different from general scene, we focus on the face features which will affect the saliency in crowd. Then we use multiple kernel learning to integrate the face feature and low-level features to detect the saliency in crowd.The main work is listed as follows:First, we introduce the related work of visual saliency from spatial and temporal characteristics, and summarize some classical computational methods of spatial and temporal saliency. Then we introduce the hierarchical motion processing in visual cortex, including the hierarchical visual pathway, modeling of simple cells and complex cells, which is the basis of bionic modeling of motion feature.Second, according to the hierarchical motion processing in visual cortex, we build a computational model of the spatio-temporal motion saliency. We adopt 3D spatio-temporal filters to encode underlying motion signals, and max-pooling operation for the coding of local features. Considering the temporal relationship of the spatio-temporal motion feature between the historical and current frames, we compute the spatio-temporal motion saliency map by measuring the difference of consecutive frames and fading factor, which can reduce the influence of earlier history frames and enhance the influence of neighboring frames.Third, we implement a tracking algorithm based on the spatio-temporal motion saliency. In order to make up for the loss of color signal in motion information, we generate the final saliency map by fusion color saliency map and motion saliency map. Then, in the framework of particle filter, we measure the correlation between the predictive state and the observation by combining the spatio-temporal motion saliency with the color histogram. As a result, the object state can be determined and tracked. Visual saliency could enhance salient object area and suppress the interference area, so as to improve the tracking performance.Last, we introduce the mechanism of saliency in crowd, and pay attention to the face feature which will affect the saliency, such as size, density, pose and occlusion. In the framework of multiple kernel learning, we use face features and opponent color features to predict the saliency in crowd. It is a basic and significant work for object behavior detection and tracking in crowd scene.