Analysis, synthesis and recognition of human faces with pose variations

Face recognition is one of the most interesting and challenging problems in computer vision. A great difficulty in face recognition is the separation of intrinsic facial characteristics from extrinsic image variations. Among the latter in 2D images are pose, illumination, and expression. Unfortunately, most past studies have provided variation-specific solutions that are not applicable to other types of variation. Performance has remained inferior to human ability and sub-optimal for practical use.; Our investigation focuses on one of these problems, which is processing head pose information in 2D images: analyzing, synthesizing, and identifying facial images with arbitrary pose. Our goal is twofold. One is to provide a simple and general framework which may be useful beyond the specific problem of head pose. The other is to improve the pose processing accuracy of previous studies by using this framework.; We propose a localized two-stage linear system which is learned strictly from sample statistics and models shape and texture information separately. Instead of using domain-specific analytical knowledge of 3D rotation in Euclidean space, our solution utilizes a simple statistical learning framework whose applicability is not limited to the problem at hand. A wider range of head poses is covered by a number of local linear models distributed over various poses, each of which realizes a continuous mapping function which directly associates a face's representation with the corresponding 3D head angle.; Our experiments prove the system to be very accurate in terms of pose estimation and image reconstruction as a function of pose while covering a greater pose range and a higher number of rotation dimensions than previous systems. The systems generalization capability over unknown poses and persons is also shown. This facilitates continuous and smooth coverage of pose variation and learning from few samples.; The main contribution of this study is a solution that is simultaneously simple, general and accurate. Its data-driven nature simplifies an otherwise labor-intensive data collection procedure. Possible applications include on-line sequential learning systems and man-machine communication.