Multilevel analysis of human body, face, and gestures with networked omni video array

Intelligent environment is a sensor array system assisting people in their working and living spaces for improved efficiency, safety, and security. In this dissertation we propose a networked omni video array (NOVA) system. The research objective of the NOVA system is to automatically derive awareness of human location, form and gesture, identity, and integrated activity through the unique semantic analysis levels. In addition, this system is unique in its processing algorithms of the semantic levels, and in its distributed multi-resolution architecture.; In the localization level, we propose a real-time 3D NOVA tracker. The unique 3D tracking capability comes from omni camera modeling, calibration, and wide-baseline 3D measurement algorithms. Upon tracking, the NOVA system captures human faces by perspective unwarping of omni videos or pan-tilt-zoom cameras. An experimental comparison of the NOVA tracker with the rectilinear array tracker proves its accuracy competence for environment-wide activity monitoring.; In the form and gesture level, we propose a novel 3D view-based gesture recognition of human body voxels. Using the distributed NOVA, body voxels of people are reconstructed in real-time. A 3D shape context then captures the spatial configuration of the human body, and its temporal dynamics are tracked by specific designs of HMM for different gestures. A gesture is decided by maximum likelihood among the HMM outputs. This 3D spatial-temporal gesture analysis is very promising in indicating 95% accuracy for 7 gestures on natural setups.; In the identification level, we propose video-based algorithms for multi-primitive face detection and tracking, face orientation estimation, and streaming face recognition. The key idea under these algorithms is to interpolate and accumulate the confidence scores of face analysis over frames. Experimental validation shows that the detection speed and accuracy performances are enhanced by the proposed schemes.; As an integrated system, NOVA derives six single and multi-person activities by the output of the localization, form and gesture, and identification levels. System attention is then derived to capture the intended human actions. Current experiments demonstrate the effectiveness of the multilevel NOVA integration. Future research directions of the NOVA system are also discussed.