Research On Sitcom-star Oriented Face Verification And Clustering For Videos

Recent years have observed the rapid development of Internet,as well as the popularity of mobile smart devices.Online video services are growing year by year on both platforms,and they also become an important part of the Internet economy.Among them,video-oriented business,including video advertising,video recommendation,etc.,focus on the interest of users,who generally pay more attention to movie stars.In the thesis,we considered the star effect of leading characters in the video,by studying face verification and face clustering for videos to facilitate sitcom-star oriented applications,including video content-based image retrieval and similar product recommendation,automatic identification of leading roles in a cast list,as well as quick video browsing and organization,etc.By comparing with static images,videos can generate large-scale face dataset based on the trait of video image sequences which show slight variations between consecutive frames.However,studies on real-time face detection and face clustering for videos have great challenges,including a variation of face posture,illumination,facial emotion,occlusion,and low resolution,etc.In this study,by considering the sitcom-star effect of popular TV serials or movies,we reviewed related works on face image processing and computer vision technologies such as face detection,face alignment,feature extraction,face recognition,and face verification.A system was implemented integrating the proposed face verification and clustering algorithms.Promising results were achieved by evaluating the performance of several state-of-the-art clustering algorithms on our constructed face dataset.Specifically,in our implementation,an open source face engine,i.e.,Seeta Face was integrated into our framework.103 episodes of “The Big Bang Theory” were collected to formulate our video dataset.Then,we extracted one frame from a fixed-length time duration to generate a raw frame image dataset.After that,we adopted face detection to get the location of the faces occurred in the video.A landmark localization module was used to detect the five landmark points,including locations of two eyes,nose,two corners of the mouth.Coordinates of the landmark points were used for face alignment in our implementation.All the detected faces were aligned to the same posture based on a pre-set standard face and normalized to fixed scale with 256x256.Similarly,a small standard face dataset,which only contains the face images of the leading characters occurred in the cast,was constructed for face verification through the same procedure.For automatically cleaning the face data,we proposed to recognize the sitcom-stars by face verification,i.e.recognition by verification,which can ease the workload of data cleaning.By designing a new verification strategy,face images of the non-protagonists were filtered out,as well as the ambiguous faces.Finally,we obtained our sitcom-star face dataset after cleaning and tagging.For video face clustering,we extracted deep CNN features from a pre-trained VIPLFace Net,and proposed a new distance measure by considering both spatial and temporal distance between each two detected faces.To evaluate the effectiveness of our proposed method,we applied our proposed distance into several implemented clustering algorithms on the constructed dataset.Experimental results demonstrated promising performance of our proposed clustering methods on several evaluation metrics.