Research Of Key Technologies In Video-on-demand Systems Based On Hadoop Framework

Benefit from its easiness to create, quickness to propagate characteristics, online media has begun to gradually replace the traditional media, and become one of the main channels for information diffusion. There is an urgent need for building a platform, which enables the user not only to upload video easily, but also play the video content on demand smoothly. The video-on-demand(VoD) systems, especially the online VoD system, can make all these possible. Since it is very extensible, able to work on heterogeneous devices, and do not require manual upgrade, online VoD websites has become one of the mainstream implementations of online VoD systems.In this work, we take the online VoD website as an example, to study the key technologies in the VoD system. We mainly focus on the video uploading, video transcoding and video storage process, introduce the current mainstream implementations, and analyze the advantages and disadvantages of these solutions. Then we make some improvement on the solutions, and propose new ones. In order to measure the actual performance for each solution, we build different modules to address the issues separately, and use simulation to compare the proposed solution with the-state-of-the-arts.For video uploading process, we propose a parallel video uploading scheme,which make use of several servers to receive the data. The test results show that this scheme is able to improve the client’s upstream bandwidth utilization, by connecting to multiple uploading servers simultaneously. In the meantime, this scheme can dynamically adjust the amount of data to be sent to each uploading node, according to the actual network conditions. By dynamically selecting the appropriate data block size, the proposed method is able to reduce the additional overhead introduced by transmitting control signals, while it can still retaining the flexibility to adjust the load on each link.For the video transcoding process, this work analyzes the characteristics of data storage in Hadoop distributed file system(HDFS), and designs a novel video transcoding task segmentation strategy. We adapt this strategy to the distributed video clip transcoding scheme. The test results show that this strategy is able to effectively reduce the aggregated load on HDFS data-nodes, by distributing the read requests to the same block in different time slots.For the video storage process, this work analyzes the characteristics of the transcoded video streams. It removes the need for creating file-level indexes, and allows us to locate a specific video clip by doing some simple calculations. In most cases, the proposed scheme is able to retrieve any video clip by reading only one data block. Even in the worst case, it only requires reading up to two data blocks to retrieve the video clip. The experimental results show that the scheme can handle post-transcoding video clips very well, and store them efficiently. If we select the load factor properly, the amount of padding data can be reduced effectively. And if the data compression feature in the underlying file system is enabled, we can even ignore the storage overhead caused by data padding.