Research On Optimization Of Transform And Quantization Arithmetic In H.264

H.264 is the latest video encoding standard established by JVT (Joint Video Team) between ITU-T VCEG and ISO/IEC MPEG. For the enhancement of the coding efficiencies, the new standard adopts lots of new techniques as followed:4×4 integer transform, spacial intra-prediction,1/4 pixel motion estimation, multiple reference frame and variable block size in inter-prediction, context-adaptive entropy coding and new loop filter deblocking. It has a significant performance benefit comparing with older standards (such as MPEG-4 and H.263) in compression performance. However, it also has a considerable increase in encoder complexity, which limits the application it can be used for. Therefore, the research on how to reduce the computations of the encoder is vital for the standard.The DCT (discrete cosine transform) and Q (quantization) are two important functions in H.264 encoding, especially in some application at middle or low rates. Since motion in this kind of application is smooth, the computation ratio for transform and quantization in encoder will increase. Therefore, it is necessary to reduce the computation in this module. So the main endeavor of this paper are to achieve maximal processing speed and better video compression quality through optimization for integer transform and quantization in H.264 encoding process and then a software based real-time H.264 video encoder is realized on the personal computer.At first, the paper analyzes the theory of integer transform and quantization in H.264, and then the Gaussian distribution is applied to study the residual coefficients and integer 4x4 DCT coefficients in H.264 encoding, according to the properties of DCT coefficients'energy distribution and characteristics of zigzag scan divide into the type of pruned DCT block is built for both luma and chroma component. Finally, the paper presents an efficient approach for prediction AZBs by ultilizing the type of pruned DCT block and the sum of absolute transformed difference (SATD) when the Hadamard transform is enabled for H.264. It is shown by experimental results that the proposed method can achieve the best performance in reducing the DCT and Q computations and obtain almost the same video quality as the original encoder in H.264.Last but not least, some more time-consuming modules such as computing SAD (the sum of absolute difference) in integer pixel motion estimation, integer transform, quantization, computing the Hadamard transform of difference matrix and computing SATD (the sum of absolute transformed difference) in sub-pixel motion estimation were optimized by using the SSE2 instructions. According to the experimental results, the speed of corresponding module has increased after optimized and achieved the same picture quality compared with the original encoder in H.264.