Synchronized distribution framework for high-quality multi-modal interactive teleimmersion

This dissertation investigates issues of performing synchronized distribution of time-correlated multi-modal continuous media data in the distributed interactive teleimmersion, and proposes approaches to solve these problems. The current best-effort shared Internet is unreliable in supporting bandwidth-savvy multimedia streaming, and frequent network jitter can contribute to the heterogeneity of distribution latencies (a.k.a., synchronization skews) within and across the multi-source sensory streams over time. The system limitations and computation constraints at the end devices can also prohibit the multimedia data in preserving the synchronization dependencies. Existing studies usually control synchronization skews by inserting buffering latencies at the media receivers. But due to the real-time nature of interactive multimedia in both two-site and multi-site applications, these approaches can impact the communication interactivity and media signal intelligibility, and affect the subjective satisfactions of end users.;To address these issues, we propose a novel and systematic synchronization framework for teleimmersion, and more generally, distributed interactive multimedia systems. We first discuss a new multi-dimensional synchronization specification model that captures the scalability of multi-modal devices, the heterogeneity of multimedia, and the diversity of the end user activities in the next-generation multimedia applications. An adaptive decision policy is also included for identifying the time correlations of multimedia frames captured from distributed media sensors. Based on the new model, we then develop collaborative multi-tier synchronization controls that consistently minimize the multimedia skews during media processing and distribution. Specifically, at the sender capturing tier, we present a dynamic computation resource provisioning algorithm to bound the skews arising during multimedia computation. At the distribution tier, we design a synchronized multi-stream distribution overlay with cooperative bandwidth management according to the diverse end user interests. At the receiver presentation tier, we perform presentation scheduling scheme to realize the multimedia synchronization at the output devices. To evaluate the tradeoff of the synchronization quality, media signal intelligibility and interactive quality, and their interactions with end user perception, we further conduct extensive subjective evaluations, and utilize the real perceptual feedback to drive the media presentation adaptation. We elaborate on the implementation of the synchronization service component in our Illinois TEEVE (Teleimmersive Environment for EVErybody) testbed. Repeatable experimental results from both the teleimmersive emulator and the real TEEVE system confirm the effectiveness of our framework.