Rate Distortion Model Based Rate Control Technique Research For H.264

With the development of video coding and network technology, video communication plays more and more important role in people's daily life and work. As the most popular video coding standard, H.264 adopts many new techniques and acquires higher compression efficiency and network adaptability compared with prior coding standards, and adapts to any kinds of video communication. Due to the diversity of video source and transmission channel, rate control (RC) becomes key step and is indispensable for any actual video coding and transmission system. Rate control is used to regulate output bitstream to meet the characteristics of channel transmission and to optimize the perceptual video quality by adjusting coding parameters under constrained conditions such as target bit rate and buffer fullness. The adoption of many new coding techniques especially rate distortion optimization scheme makes the rate control for H.264 more difficult. Research on rate distortion optimization techniques and rate control schemes for H.264 under various applications has significant theoretical meanings and actual application values. Based on the technique features of H.264 video coding, this work focuses on the rate control techniques for H.264. The main research work are as follows:(1) To address the bad effect of intra coding RC of H.264, a novel image complexity adaptive I frame RC algorithm is proposed. This work first detects the gradient of luma pixel in I frame by Sobel operator and establishes edge direction histogram for each 4×4 block, hereby gets the most probable intra prediction mode and corresponding reconstructed block, finally obtains the residual picture which is close to the actual coding residual. The mean absolute value (MAD) of residual is used to represent I frame coding complexity, then an empirical rate quantization model is proposed, and the optimal QP of I frame is determined accurately for each GOP according to allocated target bits by simultaneously considering buffer status and sequence characteristic.(2) Aiming at the shortage of H.264 classic RC proposal JVT-G012, an improved MB layer RC scheme is proposed based on the spatial-temporal correlation among basic units. First, to reduce computation cost and inaccuracy of linear MAD prediction at MB layer, MAD is computed directly according to the difference between current MB and the reference blocks pointed by estimated MV using intensive motion similarity. Then, MB header bits are predicted based on spatial-temporal correlation because MB header bit prediction has great effect on rate distortion model and QP computation. Finally, MB target bit rate is allocated according to its complexity and the parameters of quadratic R-D model are updated using coded MBs with high spatial-temporal correlation not the last coded data points.(3) To address the problem of passive frame skip occurs frequently and decoder side quality fluctuates in H.264 low bit rate applications, a RC scheme combined with adaptive frame skip is proposed to encode important frames and skip trivial frames. To get subjective friendly video sequences, the frame skip rule is based on subjective metric (structural similarity) between original frame and reconstructed frame and buffer status. The saved bits from skipped frames are allocated to key frames to enhance their coding quality, and the skipped frames are recovered from key frames to get constant frame rate and smoothed video quality at decoder side.(4) Conventional RC schemes take mostly objective metric as distortion measure, which can not acquire optimal subjective quality. This work applies structural similarity (SSIM) based subjective distortion to rate distortion optimization and RC in H.264 video coding, and proposes a SSIM optimal MB layer RC algorithm. First, an empirical SSIM linear distortion model is put forward. Then an improved quadratic rate quantization model is combined to obtain the close-form solution of SSIM optimal MB layer quantization step by Lagrange multiplier.(5) Subjective distortion (SSIM) is used to direct RDO based intra and inter MB mode decision in H.264 video coding. The research work includes two parts. The first part uses SSIM distortion to direct I frame intra MB mode decision based on the proposed I frame R-Q model and SSIM linear distortion model, and further proposes a frame layer adaptive Lagrange multiplier (λ) to balance rate and SSIM distortion better. The second part uses SSIM distortion to guide P frame inter MB mode decision on the basis of P frame SSIM optimal MB layer RC, and further proposes MB layer analyticλto trade off rate and SSIM distortion. Experiments show that SSIM optimal RC and RDO mode decision encodes image structural information better and gets higher subjective quality compared with objective quality based RC, and has low complexity and thus can be used in actual video coding applications.