| In recent years,with the high-speed development of the Internet of things,the amount of data produced is growing sharply,and network streaming media transmission as one of the important application of Internet of things industry,produce very considerable amount of data.The heterogeneity of the terminal and the instability of the network will lead to the need for diversification of the video encoding format.The storage and analysis integration of mass and multi-format streaming media is a research challenge.The traditional storage can not meet their needs,while transcoding is computationally intensive work,and centralized storage is difficult to extend.Distributed storage and computing technology is to overcome the problem.Hadoop,as open source distributed data storage and analysis tool,with its ease of use and stability has been widely used.Based on unstructured characteristics of streaming media video data,directly applying the Hadoop distributed framework to network streaming media processing will encounter many problems.This paper thoroughly studies the Hadoop framework and the characteristics of streaming media video data,and implements the massive audio and video data transcoding in Hadoop platform.Using HDFS to store mass streaming media,using MapReduce to parallelly accelerate the process of transcoding,combined with the container resource allocation and scheduling,improves the utilization rate of resources and optimizes the transcoding time.The main work and innovation points as follow:The paper proposes a video segmentation method based on the location information in Hadoop.At present most of the studies in cluster transcoding acceleration are using open source tools for audio and video pre-segmentation,which increases the additional file read and write overhead.And video segmentation processing is fixed on a node to complete,when the video processing service request is too large,the node will become a bottleneck in the system.The paper modifies the FileSplit method in getsplit function of MapReduce,obtaining the pos information in AVPacket.Through reading more data packet before the split point,avoid the segmentation in the key frames and frame jumping after video transcoding.The realization of the video segmentation in Hadoop,avoiding the performance bottlenecks,saving the overall transcoding time.The paper proposes a container resource configuration method in Hadoop Transcoding cluster based on requirements of a sample split.The proposed method solves the problem of finding the optimal configuration of container with the least expense when the cluster changes.The container resource configuration method in Hadoop Transcoding cluster based on requirements of a sample split determines the memory configuration factor and the CPU-vcore configuration factor in the sample cluster.Transcoding only one sample split and analyzing its memory and CPU requirements,and then calculating the optimal container resource configuration through the container resource configuration algorithm.The method applies both virtual cluster and physical machine cluster.The experiment results show that the optimized system,compared with the default one,is approximately 25%better in the transcoding time,mitigating the memory overload in the virtual cluster,and has a improvement in CPU usage in a certain extent. |
