Study And Optimization Of CABAC Entropy Coding And Rate-distortion Optimization In H.264/AVC

H.264/AVC is the newest international video coding standard that provides much higher compression performance than other previous standards. The reason contribute to this is H.264/AVC adopts several essential technologies, such as Context-Based Adaptive Binary Arithmetic Coding (CABAC) and Rate-Distortion Optimization (RDO). The purposes of this Thesis are to study and optimize CABAC and RDO algorithm in order to further improve entropy coding efficiency and subjective video quality.CABAC is an efficient entropy coding method used in the main profile in H.264/AVC coder that provides over 20% bit-rate reduction than that obtained in the baseline entropy coder. One essential reason is that it uses the statistical properties of the data symbols to eliminate inter-symbol redundancies. In CABAC, bit usage for coding motion vectors (MVs) accounts for a considerable portion of the bit budget. Accurately selecting context models for the encoding motion vector difference (MVD) can obtain bit-rate savings in CABAC. To realize this, an efficient algorithm for motion vector coding is proposed in this Thesis. In the new algorithm for coding a vertical MVD component, I use not only the inter-correlation between the current horizontal MVD component and the vertical one that within the same block, but also the correlation in the neighboring vertical MVD components to select an appropriate context model. Moreover, I adopt different schemes according to the encoding partition sizes. For small block sizes, I only consider the correlation among the neighboring blocks. Whereas for large block sizes, I also employ the inter-correlation between the two MVD components in the current block in order to improve the probability estimation of symbols. These strategies enhance the accuracy of the context model selection in motion vector coding, thus elevate the efficiency of the context-based arithmetic coder. I have implemented the proposed algorithm based on the H.264/AVC reference coder JM 12.2. Experimental results show that the proposed algorithm improves compression performance compared to the original CABAC scheme.RDO is another important technique used in H.264/AVC coder. It tries all the best to improve the objective video quality under certain rate constraints; in the meantime it ignores the subjective quality. However, the ultimate video quality is judged by the Human Visual System (HVS). Therefor, it is wise to adapt the coding algorithm to the sensitivity of the human eyes. This Thesis proposes a novel macroblock-level RDO algorithm based on perceptual features of HVS. Three visual distortion sensitivity models are created to minimize the perceptual distortion rather than traditional mean absolute difference (MAD) distortion. During the rate-distortion optimization process, the Lagrange multiplier is adjusted adaptively according to the visual distortion sensitivity of the encoding macroblocks. For visual distortion sensitive macroblocks, I assign smaller Lagrange multiplier so that the distortion-reduction is weighted more than rate-reduction. Better visual quality is obtained by the lower distortion in these regions with a relatively higher rate. On the other hand, rate balance is achieved by arranging larger Lagrange multiplier to macroblocks that are perceptually less sensitive to the distortion, so that more distortion is allowed without noticeable visual degradation in the decoded images. Experiments also have been conducted based on the reference coder JM12.2. Simulation results show that the subjective qualities of the decoded frames are improved without compromising PSNR.