Researches On Key-Frame Selection Method In2D-3D Video Conversion

The audience has become interested in3D movie and TV program. But the shortage of3D videos is a big problem we have to face. Currently, the3D video is generated by three ways:Original3D, CGI3D and2D-3D conversion. Because of low cost and qualified3D effect,2D-3D conversion has gained more and more attention.2D-3D conversion can be divided into three types according to the labor cost: manual conversion, semi-automatic conversion and automatic conversion. The3D effect of manual conversion is best, but its labor cost on depth assignment is extremely high. On the other hand, automatic conversion can estimate the depth map automatically, but the3D effect is not good enough. As a tradeoff, in semi-automatic conversion only the depth map of key-frame is assigned manually and its3D effect is close to that of manual conversion.In semi-automatic conversion, the depth map of non-key-frame is propagated from the depth of key-frames. And, the3D video is synthesized by virtual view rendering. However, few researchers have paid attention to the key-frame selection in2D-3D conversion. And the uniform temporal distance-based method is used mostly in2D-3D conversion because of its simplicity.To get less propagation error and high qualified3D video, we proposed a key-frame selection method based on occlusion. The occlusion has a very important influence on the propagated depth map, so firstly we calculate the occlusion of input2D video. The thresholds are set according to the occlusion to avoid the redundancy of key-frame and large propagation error. And the candidate key-frames are selected by the correspondence ratio. Lastly the candidate key-frame with minimum re-projection error is selected as key-frame to make the3D video is more realistic.A lot of time is spent on the depth map assignment in semi-automatic2D-3D video conversion. We proposed a key-frame selection method base on depth estimation. And the depth map of key-frame is estimated automatically. The candidate key-frames are refreshed by the color difference between the foreground and background, the area of foreground. Lastly the key-frames are selected according to the edge points. We don not have to assign the depth map and the semi-automatic conversion became automatic conversion.Lastly we propose a key-frame selection method based on visual attention. The frames which paid more attention are selected as key-frames. Firstly we calculate the temporal and spatial visual attention model. Then we get the spatial-temporal visual attention model by temporal priority fusion and calculate the visual attention shift by the change of attention area. Lastly the frame with larger visual attention shift is selected as key-frame.The method is evaluated by both subjective evaluation (the3D effect of synthesized video) and objective evaluation (the MSE of depth map). By comparison, the key-frame selection methods we proposed are better. The high qualified3D video can be produced with less labor cost by these methods.