Research On Segment Strategy Of Video Distributed Transcoding

Faced with the popularity of smart mobile terminals and the improvement of basic network facilities,the size and quantity of videos generated by users are increasing rapidly.Traditional single-machine transcoding services are becoming more and more difficult to meet user needs due to their slow working speed and complex expansion.With Contribution to the maturity of big data-parallel computing technology,the use of a distributed computing framework to parallelize serial transcoding tasks can achieve rapid video transcoding.The main work of this paper is as follows:(1)Based on the research and analysis of currently distributed transcoding conventional file segmentation methods and characteristics,this paper proposes an adaptive variable-length video segmentation algorithm that is more suitable for distributed transcoding based on conventional data segmentation.Adopt variable-length segment size method,set a larger segmentation window at the beginning of segmentation,shrink the segmentation window after each segmentation until the source file is completely segmented.The GOP boundary detection during segmentation also ensures that the video is parsed correctly disassociating subtasks and ensuring the independence of subtask execution.From the test result,it was found that this method has a faster splitting speed,and the number of files generated after splitting is small,thereby effectively reducing the time loss of the transcoding task in the fragmentation,transmission,and merging process.(2)Based on the adaptive variable-length video segmentation algorithm,different transcoding task scheduling strategies for homogeneous and heterogeneous distributed environments have been implemented.Each computing node in a homogeneous environment being the same,using the greedy scheduling algorithm combined with the hierarchical scheduling principle,all the fragmented files of the same segmentation length are divided into the same layer,and the high-level tasks with high complexity are processed first,this could reduce node idleness time to improve the utilization of cluster resources.In a heterogeneous environment,the computing performance of nodes is different.Firstly,the computing performance of each node is evaluated.Then,the complexity prediction algorithm based on frame type and frame length is used to predict the calculation complexity of subtask transcoding.Thirdly,the time taken for the file transport of different nodes in the distributed system is calculated and after successfully executing the three steps above,the task is scheduled to the node with the smallest estimated completion time in terms of computing and transporting through the Max-Min scheduling algorithm so that the workload of each transcoding node is kept average and the overall task completion time is shortened.(3)Finally,this article is based on Hadoop distributed development framework,Map Reduce computing model,Java programming language,combined with Spring development framework and FFmpeg video processing toolset,with My SQL and Redis database components to achieve a fully functional RESTful interface specification,The video transcoding Web system,and the system performance was tested.