Researches On Video Adaptation

The development of multimedia communication technology and cloud computing has promoted multimedia applications. However, the network heterogeneity, device diversity and application complexity has also posing new chanllenges to multimedia technologies. For video applications, the network heterogeneity requires that the video bitstream should adapt to the bandwidth and packet loss rate changing; the device diversity requires that the video bitstream should adjust to the device’s computation, storage and display capability; the application complexity requires that the video bitstream should meet the personalized user requirements and immediately response to new requests. Video adaptation transforms an inputted video to an outputted video in a new format or an augmented multimedia form to meet diverse resource constraints and user preferences. It is a promising technology to achieve Universal Multimedia Access (UMA) and Universal Multimedia Experiences (UME) and is an important research area.Video adaptation aims to bring about "anything, anytime, anywhere" user experience for video applications, it should carefully adapt to the network heterogeneity, device diversity and application complexity. The main work of this paper is invesgating key technologies according to above three constraints. First, to adapt to the network heterogeneity, we propose to extend current scalabilities (temporal, spatial and quality scalability) to error resilient scalability, with redundant picture based error resilient adaptation approach. Second, to adapt to device diversity, we propose to enable personalized and high user experience, with a Region-of-Interest (ROI) enabled adaptation framework based on H.264/SVC. Thus, enjoy high video browsing experiences even when the bandwidth and the display capability is limited becomes possible. Furthermore, since traditional "one step" approaches need a tradeoff between adaptation flexibility and complexity, and lack in intelligence and are hard to achieve UME, especially under the cloud environment, where the applications become more and more mass and intelligent, we proposes a novel cloud-aware video adaptation framework based on an intermediate video format termed Intermedia, to support new and even unknown intelligent applications and provide immediate response to a large number of concurrent users. The contribution of this paper is three-fold.1) We present a redundant picture based error resilient (ER) transmission scheme for scalable video coding (SVC) bitstream over heterogeneous networks with varying packet loss ratio (PLR). It combines different parts in video transmission, e.g. the encoder, media gateway and decoder, to achieve error resilient scalability. First, redundant picture information (RPI) is generated at the encoder under rate-distortion criterion. RPI represents whether a picture should be repeated or removed under given PLR and is transmitted to media gateway together with original SVC bitstream. Then, error resilient scalability is fulfilled at media gateway by selectively adding/removing NAL units of different video coding layers according to RPI and current network status. Finally, at the decoder, a Virtual-BLSkip and Wiener Filter based error concealment (EC) strategy is proposed to further improve the decoded video quality, which is especially suitable to conceal the loss of spatial enhancement layer. The proposed scheme scarcely affects the coding efficiency by transmitting RPI other than directly adding redundancy into original bitsteam. Meanwhile, it is able to provide error resilient scalability for SVC at media gateway with extremely low complexity.2) We propose a H.264/SVC compliant video adaptation system which supports automatically ROI tracking and efficiently ROI coding technology to enable ROI browsing function (a desirable feature when designing a video application system). The proposed framework combines both the encoder and the media gateway. At the encoder, the base layer is coded with low quality/resolution and without ROI slice to provide basic video quality for devices with low bandwidth or small screens. The spatial enhancement layer contains ROI slices and provides higher video quality for devices with medium/high bandwidth or small/large screens. In the proposed system, ROI is first tracked with proposed particle filtering based tracking algorithm with considering base layer motion information, then, proposed rate-distortion optimized mode decision method, which improves the coding efficiency by relaxing the temporal constraints while taking into consideration the mismatch between reference frames when Background slices are discarded or kept, is used to encode ROI slice. User is allowed to choose the desired ROI when the network bandwidth or the screen size is limited, and the media gateway responses such request and extract the corresponding bitstream. In such a scenario, the proposed framework can preferentially guarantee a high quality ROI area and have the ability to maximize the user experience.3) We propose to use Intermedia for video adaptation in cloud; we design and implement a video adaptation demo system and demonstrate the feasibility and effectiveness of Intermedia as a cloud-aware media. Intermedia is an intermediate video format that consists of both signal level and semantic level descriptions. The signal level description is used to quickly generate suitable bitstream that meets the contrants of various devices and different networks; the semantic level description is used to adapt to divers or even unknown application requirements. Intermedia is pre-generated and organized appropriately, and it can be stored in or transmitted to the media gateway. Specified application may introduce several constraints, and the target bitstream can be quickly generated from Intermedia with very low complexity. Intermedia based video adaptation framework shifts the complexity from transcoder to pre-processor, thus it has the ablity to simultaneously support a large number of concurrent users. The demo system supports various adaptation operations, such as bitrate adaptation, framerate, resolution adjustment, and video summarization, etc. and supports demands from different types of clients (e.g. PC, PDA, etc.) connected to the server through different networks.