Optimization Of Grid-Based Industrial Data Transfer

Withing the usage of grid computing, large-scale industrial real-time data analysis is become more and more important for industrial production. However, in this place, there are also having some big problems. First, the speed of real-time data generation is very fast, our system needs much more storage space, it is a big problems that how can we use the less space to store them. Second, in processing stage, thousands of links will exist between agents and our server, the stability of our system is also very important. Third, the traditional TCP solution is no longer suiltable for current high speed network. In this paper, we will doing three aspects for our systems, data compression, the grid server performance performance and the network transmission performance optimization. This paper mainly completed the following works:(1) As time increasing, industrial data will produce a gradient and form a line which changes slowly. The size of data produced by industrial machine is very big, and it is increasing every seconds. We cannot buffer a large number of the data, then analysis them to choose a suitable compress solution. The compressing solution must be with high compression ratio and fewer resources consumed. In this paper, we present a compression algorithm called curve area mapping algorithm. The compression ratio of our new algorithm is2.16times than traditional SDT algorithm.(2) Server performance is one of the critical activities in the data grid environment. A large number of applications require access to huge volumes of data from grid servers. In this case, efficient, scalable and robust grid server which can deal with large file transfer concurrent is needed. In this paper, we analyze the bottleneck of our grid servers and introduce user-space I/O scheduling, zero copy and event-driven architecture in our grid server to improve the servers’ performance. The user-space I/O scheduling can save almost50%I/O time in a huge number of small files transfer. Grid servers can elimination CPU consumptions between kernel and user space by zero copy and cut63%times for context switches. Event-driven architecture will save30%CPU usage to reach the best performance by thread-driven architecture. Optimization method combination of these three above are used in our grid servers, the full-load throughput of our system is30%more than traditional solutions and only60%CPU consumed compared with traditional solutions.(3) Network traffic test shows:95%of current network are TCP stream flows, the other is using UDP or other protocol data streams. Therefore, the performance of TCP protocol has become the key of the whole data transfer performance in entire network. By analysis the TCP protocol and its variants, comparison the different congestion control algorithms of them, we discussed the performance of these TCP protocols in different network environments. Achieve the optimizations for TCP protocol which is widely used. Present a BP network predict algorithms used to predict Packet loss rate. By the test, the algorithms can reach74%correct rate.