| Distributed computing framework use cheap PC for building multi-core cluster to solve the problem of large-scale parallel computing, but programming on Distributed computing framework is still a very complex task because of the need to consider load balance, communication and synchronization overhead. As a domain specific programming model, data flow programming combines the features of media applications and programming languages and offers an attractive way to express the parallelism. However, hierarchy storage structure of multi-core cluster architecture puts forward new challenge for the performance of data flow application. Besides, the programmability remains a significant challenge for the compiler.Aiming at the data flow programming model in the face of big data processing of digital media field problem, design and implementation of the integration of data flow programming model and the distributed computing framework —— proposed a compiler and optimization framework for Storm based on COStream. Compiler optimization method for Storm includes two steps: hierarchical task partitioning and scheduling for Storm, pipeline scheduler and code generation for Storm. hierarchical task partitioning and scheduling is used to assign the task to the multi-core cluster nodes within the cluster, to ensure the workload balancing between multiple cores, inter cluster communication overhead is small. The pipeline scheduler and code generation are used to build software pipeline between cluster nodes and build software pipeline between cores in node, and generate the corresponding object code.The experiment on multi-core cluster for the target platform, and build Storm distributed architecture in the cluster, select digital media processing typical program as the benchmarks, evaluated and analysis the performance of optimization for Storm?The experimental results show that effectiveness of combined with the distributed computing framework and data flow programming model. |
PDF Full Text Request