Fast Mode Decision Algorithm For H.264/AVC

H.264/AVC is a codec standard developed by the ITU-T Video Coding Experts Group (VCEG) together with theISO/IEC Moving Picture Experts Group (MPEG). H.264 employs a number of new technologies which contribute to its better coding performance compared with previous standards. However, these new technologies also lead to the dramatically increased computational complexity, which makes it difficult to use the H.264 standard in the systems with limited computation abilities or in the circumstances where real-time transfer is required. This is the reason why fast encoding algorithms are needed.The most complex and time-consuming part in the H.264 encoding process is the mode decision. H.264 employs rate-distortion optimization method to decide which mode to be used to encode a macroblock. For intra blocks, it is necessary to calculate the rate-distortion cost for 592 times in order to find the best intra mode; For inter blocks, more computation is involved, including motion estimation, motion compensation, transform, quantization, entropy coding and so on. The mode decision process takes most of the overall encoding time. Therefore, optimizing the mode decision process is of great importance to reduce the computational complexity and will bring significant performance enhancement for the encoder.In this thesis, we first introduce the structure of H.264/AVC encoder and the key techniques the encoder employs. Then the mode decision process is studied, which includes the criteria of mode decision—rate-distortion optimization and the three mode decision methods in the JM reference software. After that, we discuss the main methods used in fast mode decision algorithms, including approximating the cost function, reducing the set of tested modes and early termination. Following that, a typical fast mode decision algorithm, the reference algorithm in this thesis, is studied and implemented. Several imperfections are revealed by analyzing the experiment result. Then a new fast mode decision algorithm is proposed with three techniques employed to optimize the reference algorithm. The three techniques are namely macroblock adaptive threshold design, optimized mode grouping method and mode decision using luminance residual information. The result of the experiment indicates that the proposed algorithm would lead to a much lower bit rate than the reference algorithm while PSNR and total coding time are almost the same.