Video Transcoding And Optimization Of Heterogeneous Distributed Clusters

Video is constantly being produced and used in more presentation formats,device types,and a variety of networks environment.Video transcoding is the process of converting a video encoding format to another video encoding format.However,most of the time,transcoding is a computationally intensive process.Therefore,people use distributed computing technology to efficiently use the available computing resources in multi-machine,multi-core CPU and distributed computing resources in a specific facility,home or private distributed infrastructure to handle complex task.In a distributed cluster,video is splitted into multiple segments and parallel transcoding is implemented on multiple machines.Hadoop is a very popular programming framework in distributed computing technology.It provides high scalability for distributed systems and meets the high scalability requirements of video transcoding.In a distributed cluster,the computing power between machines is not necessarily the same,and the heterogeneity of computing power is a very common phenomenon in distributed systems.In this paper,we study the video transcoding acceleration of heterogeneous distributed clusters from the perspective of task scheduling and system architecture.The main works and innovations are as follows:(1)The unbalanced load of heterogeneous clusters makes the cluster's use of computing resources unreasonable.As a result,the entire finish time of video transcoding job is much higher than the ideal value.Therefore,a load balancing task scheduling algorithm can achieve distributed video transcoding acceleration.The model of Max-MCT and MLFT task scheduling algorithms does not consider the transmission overhead of video segment,resulting in the task scheduling model inaccurate.The PLTS algorithm considers the segment transmission overhead but does not balance the segment transcoding time and transmission time.Therefore,the expected completion time of the job still has room for optimization.From the point of view of MapReduce task scheduling,in order to effectively use the heterogeneous computing resources of the cluster,in this paper,we first the Hadoop video transcoding task scheduling model.In order to optimize this model,we turn it into a solution to the NP-hard problem and propos a heuristic algorithm called LA-MCT,the main idea of this algorithm is to balance the video segment transcoding time and the segment transmission time in the cluster.A lot of simulation experiments show that our algorithm has shorter job finish time than the existing heuristic task scheduling algorithms such as Max-MCT,MLFT,and PLTS algorithms have shorter job finish time.(2)From the video transcoding architecture point of view,in order to speed up the running process of the entire transcoding system,we abstract the entire system's operation flow.Alluxio distributed memory file system is used instead of the existing HDFS file system to implement video buffering and sharing within the system,which reduces the disk read and write overhead of video fragmentation in the video transcoding system.We have built a small Hadoop heterogeneous cluster,designed and implemented a video transcoding system,the system uses different sizes of video data,and many experiments have proved that the Hadoop heterogeneous cluster based on Alluxio outperforms the existing HDFS-based Hadoop.The speed of a video transcoding job increased by about 5%.