Research On Task Partition Optimization And Data Transmission Method Optimization Of COStream Data Flow Program

COStream is an efficient data stream programming language.By separating communication from computation,it highlights the high parallelism of the program,which can take full advantage of the performance of multi-core platform.However,in the task partitioning stage,the existing COStream partitioning algorithm is not universal,and the boundary of each partitioning stage is fuzzy.In terms of data transfer,the buffer design at the bottom of the compiler is not systematic enough,and as a language that supports parallel programming,the thread synchronization mechanism of COStream also has room for optimization.Aiming at the existing problems of COStream task division,the task division was divided into two stages for optimization.In the initial stage,a hash division algorithm based on the specific hierarchical attributes of COStream computing nodes was proposed.In the node movement stage,load balancing and communication overhead are optimized respectively to restrict the type and movement direction of candidate nodes in the maximum load subgraph for load balancing optimization,and to expand the type range of candidate nodes in the gain structure for communication overhead optimization.At the same time,the gain structure is introduced to reduce the repeated calculation of each iteration of the algorithm,the tabu list is designed to ensure the correct convergence of the algorithm,and the perturbation strategy is adopted to enlarge the search range of the solution.Aiming at the problem of data transmission,the paper designs and implements the tail-node buffer,completes the COStream buffer system,and solves the problem of data transmission between the tail-node and the hard disk.The thread synchronization mechanism of COStream is improved.For the thread synchronization in static subgraph,the global variable array is used to simulate the barrier mechanism,and the data transmission between nodes is optimized with the idea of lock-free programming.Eight classic applications in multimedia field were selected as test programs,and the improved task partitioning algorithm,tail node buffer and the improved thread synchronization mechanism of COStream were tested.The experimental results show that the optimized COStream has a certain improvement in program performance compared with the old COStream.