Lightweight Image Super-resolution Model In Live Video Streaming

Image super-resolution is one of the research directions in computer vision,and it is applied in some fields.Previous image super-resolution mostly focuses on the reconstruc-tion quality; for live video scenarios,image super-resolution needs to focus not only on the reconstruction performance but also on the inference speed,and thus the lightweighting of the model becomes an important research.In addition,the image pixels increase expo-nentially after image super-resolution reconstruction,and the bit rate generated by video coding also increases,which increases the network bandwidth cost.Therefore,reducing the bit rate of images and videos with no or less visual loss is also a factor to be considered in image super-resolution applications.For the image super-resolution problem in live video broadcasting,this paper mainly does the following work.(1)Study the lightweight implementation of image super-resolution model and pro-pose a lightweight image super-resolution model based on feature dimensionality reduc-tion.The model contains a lightweight downsampling module,and a Pixel Unshuffle-based feature downsampling module is added to the head of the model to reduce the feature scale without losing features basically,in order to reduce the computation of the network and achieve the purpose of fast inference.In the training phase of the model,a loss func-tion based on high-frequency enhancement is proposed to guide the network to enhance the sensitivity to high-frequency textures during the learning process of the model to ef-ficiently reconstruct distorted detailed textures in images.A dataset construction method based on video compression noise simulation is proposed to maximize the image degra-dation in real scenes by imposing both space-domain and time-domain compression noise on the images.(2)Study the code-rate saving method of super-resolution in live video application scenarios,and propose a lightweight code-rate saving model based on feature low-rank reconstruction.A loss function based on feature low-rank reconstruction is designed in the training phase,and the method removes the visually insensitive noise details in the image.The model effectively reduces the rank of the image,thus reducing the encoded video bit rate with no or less visual loss in the application scenario of video super-resolution,and saving the cost of super-resolution in the live broadcast system.(3)Based on the above model and method,a live video streaming system is designed and implemented.Through the deployment of the model in the video transcoding sub-system,the live video streaming system has the ability of super-resolution reconstruction and bit rate saving,which solves the problem of not being able to watch the live super-HD video streaming due to the low quality of the original video stream at the pushing end,and verifies the effectiveness of the model and method proposed in this paper.