Research On Key Technologies Of Parallel Optimization For Multi-computing Platforms For Large-scale Applications

The emergence and rapid development of high-performance computers have made them widely used in cloud computing,security,and big data processing.According to statistics,big data processing accounted for 46% of the total.High-performance computer has many characteristics,for example,the complexity of the storage structure,the diversity of computer architectures,and the big data processing problems are large and complex,etc.However,all of these made the application of high-performance computers in the field of big data processing face enormous difficulties.This paper mainly studies the heterogeneous parallel algorithm and optimization technology of different application scenarios under various storage structures.What's more,it selects three typical application problems of iterative algorithm,high throughput requirement and large-scale network fusion in big data processing respectively.Parallel algorithms and optimization techniques for different types of application scenarios are studied in terms of storage,communication,task partitioning,parallelism,matrix vector operations,and CPU+GPU heterogeneity.The main innovations of this paper are summarized as follows:(1)A parallel SNF algorithm based on multi-level storage is proposed.Aiming at the similar network fusion algorithm(SNF)with large sample size and high memory demand in biomedical field,this paper proposes a parallel storage-based CPU+GPU heterogeneous parallel optimization algorithm para SNF,which improves the algorithm by matrix/vector partitioning.The cache hit rate;greatly improves the scalability of the SNF algorithm by adopting a three-level storage model based on SSD+memory+cache.The experimental results show that the para SNF algorithm has fast operation speed and high scalability.(2)A heterogeneous parallel fingerprint matching algorithm with no data correlation and high throughput is proposed.Considering the increasing size,high real-time requirements,continuous improvement of recognition algorithms and high data concurrency of the database,this paper takes fingerprint recognition as an example and proposes an optimized fingerprint identification system framework.For the hot issues of fingerprint matching in the system,the heterogeneous system of multi-core CPU+nuclear GPU is selected,and the matching process is optimized from the storage order of fingerprint template,task scheduling,CUDA Stream and so on.The experimental results show that the fingerprint matching algorithm based on CPU+GPU is fast and has high throughput,which can meet the real-time requirements of the system.(3)A distributed storage iterative parallel optimization algorithm based on asynchronous protocol communication is proposed.In this paper,the SBA(Sparse Bundle Adjustment)algorithm in large-scale 3D reconstruction is taken as an example.In view of the large data size,large computational complexity and high storage requirements of the SBA algorithm in the iterative optimization process,a problem independent of the BA problem is proposed.Distributed task allocation scheme,and multi-core parallel optimization of key steps in the algorithm,a distributed DSBA algorithm based on asynchronous protocol communication(A-DSBA)is proposed to solve the problem of equations in the problem.A large number of experiments showed that the proposed algorithm has high scalability and fast computing speed while maintaining the accuracy of the algorithm at the same time.