Research On Three-dimensional Video Enhancement And Video Retargeting Technology Combined With The Visual Saliency

Three-dimensional video can bring more realistic 3D feeling to people. However, because of the immature of the technology, there are many problems. To improve the 3D video technology, this paper conducts the research from two aspects which are 3D video enhancement and video retargeting. We can improve the quality of the rendering 3D video by means of 3D video enhancement. Using the 3D video retargeting technology, the 3D video can adapt to different screen size,but this technology requires to protect the important area of the video to avoid the visual distortion. So this paper first conducts the research to the visual attention and proposes two video saliency detection model to detect the important area of the video. Then, according to the 3D video enhancement and video retargeting technology, this paper proposes the feasible methods. The specific research contents include the following aspects:(1) To deal with the problem of the visual attention modeling,we present two video saliency detection model. In the first model, we sample the input video sequence into three planes: X-T slice plane, Y-T slice plane, and X-T slice plane. Then, motion saliency map is extracted from the X-T and Y-T slices, and static saliency map is extracted from the X-Y slices by low-rank matrix decomposition. Finally, we combine these maps to obtain the video saliency maps. In the second model, we extract the DCT coefficient and motion vectors from the video bit stream, which are used to produce the texture saliency map and motion saliency map, respectively. After that we quantify background complexity to modulate the texture saliency map and combine residual frame with the motion saliency for optimization. Finally, we use the saliency uncertainty to decide the weight of fusion.(2) On the research of 3D video enhancement, this paper proposes two methods for video enhancement,to deal with the depth image based rendering technology, we present a depth map post-processing method for virtual view synthesis. In the method, the filtering window size for each pixel is first established based on the intersection of confidence intervals rule. Then, the depth value for each pixel within the filtering window is estimated by using least square polynomial approximation. Finally, a weighted mode filtering is applied to the estimated depth map, and obtains the filtered depth map.To deal with the low resolution depth map obtained by the depth data acquisition system, this paper presents a depth map reconstruction method based on the intensity and depth co-sparse analysis model. In the method, we use the training method to build the intensity and depth co-sparse analysis model. Then we use this model to reconstruct the low resolution depth map to create the high resolution depth map. Experimental results show that, the proposed methods can accurately recover the depth information and promote the performance of virtual view synthesis.(3) On the research of 3D video retargeting technology, this paper presents a 3D video retargeting method based on the content. In the method, we use the saliency map combined with the depth data to extract and protect the important content of the 3D video. Combined with the temporal and spatial correlation, the retargeted video is continuous in the temporal and spatial domain. Then we combine the three features together to gain the total energy map. Based on the total energy map we can find the region with less energy is less important. Then we can select the minimum energy region as the seams and remove from the video. Experimental results show that, the proposed method can protect the important content, and the result is continuous in the temporal and spatial domain.