The Research On Data Placement And Schedule Scheme In Large-Scale Multimedia Storage System

With the rapid development of computer and multimedia technology, the research on large-scale storage system is the focus of multimedia server design. Large-scale Media Server requires high storage capacity and high I/O throughput. It emphasizes much on reading performance. Data placement scheme is the key point that affects the system performance and the corresponding scheduling policy. It is important and meaningful to investigate data placement and the scheduling policy for storage system of large-scale video server, which is researched in this thesis.First, we designed a distributed soft Redundant Array of Independent Disks (RAID) in server-less clustered server. It is a clustered storage system that uses a large number of small magnetic disk drivers (MDDs): All distributed local disks can be glued together to form a big storage pool with Single System Image (SSI). It can have large storage capacity, providing high efficient disk file sharing among different nodes and high performance for media service.Second, based on the above storage architecture, we devised a data striping scheme that separate parity blocks and data blocks (SPDB) on disk region to provide the media server with high concurrent streams, and the Node-Degree Parity Group (NDPG) redundant scheme is proposed to tolerate both single point of disk failure and single point of node failure with comparatively low storage overhead. It has far better reliability than simple RAID-5 parity scheme. We analyzed its availability with Markov model, and the result show that the Availability Improvement Factor (AIF) of NDPG has 99.59% and 89.05% improvement than simple RAED-5 and RAID-x respectively.System scaling is the main approach to meet the increasing of both capacity and bandwidth, which will cause the redistribution of the whole data blocks in the system. The Reorganization algorithm should follow three rules. First the load of the data blocks should be kept balance across the whole system after reorganization. Second, the process of the scaling should have as little workload as possible. And third the algorithm should be simple and its operation overhead should be small. Assuming random placement of data blocks on multiple nodes, and do not consider the parity placement at here, We propose an efficient algorithm, called Serial Number Reorganization (SNR) algorithm, to meet the objective.An efficient measure to alleviate I/O bottleneck is to adopt multiple network-connected parallel-working disks, where I/O request can be split into several small disk requests and each disk serves one. In order to make better use of multiple disks' parallelism to decrease the request execution time, optimal concurrency degree, that how to split I/O request, is most important. It's valuable to acquireI/O request optimal concurrency degree by a mathematical analytical model because of simplicity, comprehension and rapidness. Concurrency degree is computed according to the characteristic of video requests, request size, disk data preparing time distribution function, disk bandwidth and disk I/O request administrate overhead. We have programmed a simulator to emulate disks parallel working. Using this simulator, we have done detailed verification on analytical results.In order to optimize the load balance and intra-request concurrence of storage system, we first present a strategy of balanced hypergraph-based adaptive data placement; and then bring out two objective functions for the optimization of both load balance and intra-request concurrence. It can adaptively optimize the storage performance by I/O access pattern. Finally we have done a lot of simulations to evaluate our method and compare it with traditional optimization strategies, such as adaptive load-balance and hyper-graph placement for intra- request concurrency. The result shows that our method is better than these schemes.In the end, based on the storage system with SSI described above, and the video data are striped across all server nodes and their disks, this paper proposed a server-push scheduling strategy that based on best-effort transmission, called BECP (Best-Effort Concurrent Push) scheduling policy. Different to traditional video servers schedule policy, all server nodes in the cluster send out the media data blocks as soon as they are retrieved from disks without long time buffering at server end, and the data transport and communication overhead are small. We made an analysis of the system performance and a simulation process.