The Design And Implementation Of Distributed Transcoding System

With the audio and video sharing website,webcast website,large-scale short video sites,audio and video compression and processing demand more and more,audio and video processing technology is also more and more attention at home and abroad,video sharing websites such as bilibili Youku Letv iQiyi Tencent video live video sites such as Douyu TV short video App such as Toutiao Douyin Kuaishou,single traditional transcoding,special equipment transcoding,increasingly unable to meet the existing audio and video broadcast,video website,short demand.To solve these problems,the video site in order to meet the selfproduction video and UGC users meet the production demand and internal PGC,developed a distributed transcoding system,using multiple devices in parallel transcoding,to improve the video production speed,improve the user experience.A distributed transcoding system currently using distributed computing framework Hadoop framework for distributed video transcoding,the documents and other data processing methods are different,so the common open-source distributed computing framework for multimedia document processing is very difficult.In view of this problem,this paper analyzes the advantages and disadvantages of the general open source framework,and proposes further improvements.The main innovation points of this paper are embodied in:(1)for process optimization of task scheduling strategy,users overall the shortest waiting time,according to the IO and computational tasks are split and reasonable planning and scheduling for multimedia files,IO optimization,distributed transcoding is Hadoop open source framework for the reduction of the 4 IO operation,the redundant IO file copy operation is reduced to a minimum.The maximum use of machine resources to improve the transcoding speed.(2)the whole process of distributed transcoding can be monitored,measurable,and in stages,and the time-consuming historical data can be traced back.With cross cluster monitoring management system,it can control the task ratio between cluster and single cluster switch,which is convenient for cluster load balancing,cluster operation and maintenance and gray level.(3)The use of distributed monitoring and coordination technology,any node failure does not affect the work of the cluster.The scheduler will automatically remove the failure node from cluster,and job on it will be rescheduled to other idle node,any job stuck will not affect the cluster work.The scheduler sets the timeout period of the job and will reclaims and reschedule the overtime job.Even if the scheduler instance is faulty or the scheduler exits abnormally,it will not affect cluster work.Multiple scheduler instances will automatically elect the master node and the master node will resume work.The experimental data show that the single machine with the length of the video transcoding time increases linearly with the length of the video transcoding time distributed growth is not obvious,the transcoding system has been in production environment is stable,deployed in hundreds of machines,the average daily throughput reached tens of thousands of video level.Experimental data show that in the performance test,the single transcoding timeconsuming with the linear increase of video duration,distributed transcoding time-consuming as the video length of the growth is not obvious,36 minutes of video distributed transcoding than single transcoding less time-consuming 69%.It can be seen that the transcoding system can significantly reduce the time-consuming of transcoding as the video duration increases.During the waiting duration test,10 videos are concurrently submitted for transcoding.The waiting time of the users in the distributed transcoding cluster with two machines was 57.2% shorter than that of the single machine,and the waiting time of the users in the distributed transcoding cluster with three machines was 19.4% shorter than that of the two machines.This shows that as the number of users submits a large number of videos,the average waiting time for users can be significantly reduced as computing nodes increase.Currently,this transcoding system is used in the production environment of a large video sharing site and is deployed in a few 100 machines,the average daily video throughput reached tens of thousands of levels.