Research And Application Of Enhanced Video Coding Based On Deep Learning

With the development of the Internet,the demand for video applications has grown rapidly and been more diversified,bringing tremendous pressure on network bandwidth and storage.Even with the current state-of-the-art coding standard HEVC,it is difficult to achieve sufficient compression efficiency in many cases.At the same time,the deep learning algorithms — especially the convolutional neural networks — have developed rapidly since 2012 and achieved remarkable results in many fields including image processing and classification recognition.How to use the deep learning tools to improve the compression efficiency of the encoder has become a research field attracting wide interest.On the other hand,the traditional coding framework has strict restrictions on the form of coded video and cannot adapt to emerging application scenarios such as virtual reality and interactive video.How to adapt to more application scenarios using extended coding standards such as scalable coding is also a subject worthy of study.Based on the above background,this article will focus on how to enhance conventional and scalable video coding standards with deep learning techniques such as image restoration networks and super-resolution networks.At the same time,a program example is used to explore how to flexibly apply video coding technology to special scenes such as panoramic videos.For conventional coding standards,we herein propose an efficient video coding system with online training neural network.A frame restoration convolutional neural network(FRCNN)is trained for each group of pictures of each sequence to repair the quality of reconstructed frames.Using only the current encoding video stream as a training set,the FRCNN can restore the reconstructed frames very meticulously.Even at low bit rates,the final output of the FRCNN can improve the video quality effectively.Moreover,an efficient parameter coding scheme is applied to compress the parameters of the online training FRCNN.Subsequently,the compressed bits are transmitted to the decoder as part of the encoded bitstream.Compared with the latest High Efficiency Video Coding standard,the proposed system can achieve 3.8–14.0% Bj?ntegaard-Delta rate reduction,which is much higher than most of the existing neural-network-based video coding systems.The restoration network will be an additional part of the traditional standard codec without any structure change,thereby rendering it compatible with the existing coding systems.Similarly,we propose a repair & super-resolution convolutional neural network for the spatial scalable video coding.The designed network is composed of a repair subnetwork and a super resolution subnetwork,which are used to improve the quality of the reconstructed base layer and interpolate the pixels for the interlayer,respectively.In replacement of the interpolation filters in the spatial scalable video coding,the network could provide an interlayer reference with more accurate pixels and thus improve the coding efficiency.Experiments show that with the implementation on Scalable High Efficiency Video Coding,the proposed design can reduce up to 40% bit rates of the enhanced layer,and averagely 5.3% bit rates of the total stream.Based on the enhanced scalable encoder,we also designed an efficient online coding solution for the panoramic videos to demonstrate how video coding could work in emerging video applications.