Research And Implementation Of An Intelligent Inpainting System For Image Frames Oriented To Live Streaming

With the rapid development of Internet technology,short video live broadcasts have begun to penetrate into people's livelihoods and become an indispensable part of people's work,study and life,including scenes such as video conferences,live broadcasts,and live shopping.People's requirements for live broadcast quality have become higher and higher,and the vigorous construction of network infrastructure is still difficult to avoid the emergence of weak network coverage with insufficient network signal coverage or weak coverage.In addition,the high concurrency during live broadcast also puts tremendous pressure on the bandwidth and throughput of enterprise equipment.Both of these situations will cause the problem of live broadcast uplink packet loss,which will cause image frames in the live broadcast to be blurred and affect user experience.In order to solve the above problems,different streaming media transmission protocols have been proposed one after another,the purpose is to improve the packet loss situation by adjusting the congestion and retransmission mechanisms at the transmission layer.This method can solve enterprise-level problems to a certain extent,but cannot fundamentally resist the weak network environment.There are also some more general methods,such as choosing to discard related frames or even the entire GOP after packet loss is found,but this will bring a freeze to the live broadcast and also affect the experience.With the increasing maturity of deep learning in various fields,the use of deep learning to repair damaged pictures and videos has also become a hot and key direction of academic research,and certain progress and results have been achieved.This article attempts to combine the packet loss repair problem under weak network live broadcast with the deep learning video inpainting problem.First,this paper proposes a two-stage video restoration algorithm:the first stage uses 3D convolution to obtain the time domain information of the live video,and downsampling reduces the model parameters;the second stage is guided by the time domain information,combined with the spatial context information of the video repair the lost part of the image and gradually restore it to the original pixel.These two stages ensure the efficiency of the algorithm and meet the needs of live broadcast real-time.Secondly,the introduction of a learnable area regularization method to the video restoration algorithm improves the regular drift problem caused by the traditional regularization method due to the influence of the damaged pixels of the image,thereby improving the accuracy of the restoration to a certain extent.Then,referring to the idea of difficult sample mining,the repaired edge pixels are regarded as difficult samples,so as to avoid the blurred part of the repaired edge caused by the majority of smooth pixels in the training process.Finally,set up a live streaming media system,the entire system includes a live upload and download client,a streaming media processing server,and a video repair processor.This system can simulate the entire live broadcast process and at the same time simulate a weak network environment,verifying the real-time and accuracy of the proposed video inpainting algorithm.