Research On High Performance Storage System For Large Scale Streaming Service

The most important factors that influence the performance of the streaming server platform include: streaming server's maximum throughput, streaming server's Internet connect speed, and streaming storage server's maximum throughput. Content Delivery Network and cache server transfer streaming service's access point from the streaming server to the cache server which is deployed in the area near the client. This mechanism can reduce the negative effects on the quality of streaming service, which is caused by the fluctuating of the bandwidth of the network, and also can reduce the requirements to streaming server's Internet connect speed. Streaming server's load is reduced by P2P data sharing mechanism introduced among streaming clients. Thus the requirements of streaming server's maximum throughput and Internet connect speed are reduced too. Although the streaming storage system is one of the most important components in a streaming server platform, there are relative few researches on it. The soft real-time characteristic of the streaming applications makes a significant difference between the performance indices of the ordinary high performance storage system and those of the streaming storage system. In general, the ordinary high performance storage system cannot get best performance if it is used directly in the streaming service. So it is necessary to design a special high performance storage system according to the characteristics of the streaming applications.In this thesis, the architecture of the storage system for Internet-based large scale streaming service is studied. The goal is to provide the streaming storage service with high performance, the support of the QoS ensuring mechanism, and affordable system investments. The main results are summarized as follows:1. The shortcomings, which occur when popular streaming system architectures and high performance storage systems are used directly in the streaming service, are analyzed in detail. Based on this analysis, the importance of designing a new high performance storage system for streaming service is pointed out and analyzed in detail.2. The load characteristics of the concurrent streaming traffic are analyzed through system simulations. Based on this research, the relationship between the performance of the storage system and several factors, including cache size, data size and disk bandwidth, is studied, and then the design rules for the storage nodes are obtained.3. A high performance storage system based on the stripped data storage and the concurrent data access mechanism is proposed. By introducing the parallel service ability in metadata server, the performance of this system is superior to other existing systems.4. A three-level directory search method is proposed to improve the index performance. By introducing the fast table mechanism in metadata server, this method improves the index performance of hot metadata query service. Also, this method introduces the metadata prefetch mechanism in metadata query to reduce query frequency in the film service procedure. Besides that, this method uses the metadata cache mechanism to prevent repeatedly querying the metadata information of the same data object in the storage client. Thus the metadata server's cost is reduced.5. In order to further improve storage system's performance, the distributed cache layer between the streaming server and the storage system is introduced. By uniformly managing all storage space of the nodes in the distributed cache layer, a global visible cache pool is built. This mechanism improves storage system's performance greatly, especially for data with hot access frequency.6. A mechanism which integrates variable QoS differentiated services model, access control, and the QoS ensuring mechanism of MDC coding method is proposed. It can provide the ability of ensuring QoS for the streaming storage system.7 According to the requirement of the service provider to distribute video file to different areas in Internet, a data cooperative distribution mechanism based on optimum distribution bandwidth allocation policy is proposed, which is suitable for the fluctuating of Internet bandwidth. By cooperatively providing data to other destination nodes, it can improve video file's distribution performance and reduce the load of the source node..