Cloud-based Image Sparse Representation Algorithm Distributed Parallel Optimization

With the development of society,there are more and more image-based applications,such as face recognition,hyperspectral image mineral exploration,environmental monitoring and so on.At the mean time,the resolution and the size of images are increasing explosively.Sparse representation of the image is a very effective image processing method,which uses a small number of non-zero elements in the sparse representation coefficient to represent the image,thus it can provide convenience for the subsequent image analysis and application.However,due to the diversiform information of image types and the high complexity of sparse representation algorithms,it is difficult to analyze large-scale images on the existing stand-alone computing platforms,and the execution efficiency is low.Cloud computing is a scalable distributed parallel computing framework and a platform for storing large-scale data which has powerful computing power and wide application prospect.Based on the MapReduce computing framework of Hadoop,HDFS,Spark system structure,task scheduling and RDD,combined with the practical application background of image denoising,this paper designs a image sparse representation of the K-SVD algorithm based on spark and the distributed parallel optimization method of the sparse-TV denoising algorithm for hyperspectral images.And we used a large amount of image data to verify the experiment.The experimental results show that the sparse representation of the distributed parallel optimization method can improve the processing speed and the capacity of processing the large-scale data while guaranteeing the correctness of the processing results.The main contents include:1.Based on Spark cloud computing platform,the distributed parallel optimization of image sparse representation K-SVD algorithm is carried out.Based on the analysis of the K-SVD algorithm and the OMP algorithm,combined with the Spark task scheduling and the MapReduce computation framework,we improved the dictionary updating method of the K-SVD algorithm by updating the atom separately,thus it can increase the parallelism of the algorithm.According to the sparsity of coefficient vectors,a triplet structure is designed to record each sparse vector,which compresses the data size and reduces the amount of data transmission and redundancy computation.During the solution process,the OMP algorithm produce the residual vector and K-SVD dictionary need to calculate the error matrix in the process of updating.It can reduce the computation of the error matrix by optimizing the calculation of the error matrix.At the same time,a reasonable intermediate data structure is designed and making full use of the task scheduling strategy of data locality to reduce the data transportion between the nodes and the shuffle in MapReduce task.The accuracy and acceleration of this method can be verified by comparing with the single serial experiment.2.Based on Spark cloud computing platform,the distributed parallel optimization of Sparse-TV denoising algorithm for hyperspectral images is proposed.Based on the group sparse algorithm,TV denoising algorithm and PCA algorithm based on hyperspectral image,this paper combines Spark task scheduling and MapReduce computation framework to improve the PCA algorithm and reduce the computational cost.By using the parallel network transmission capability among multiple nodes,it can reduce the data transmission time during the denoising process.At the same time,designing a reasonable intermediate data structure and utilizing the locality of RDD data to reduce the data transmission between different RDDs in the TV algorithm.By sharing memory space,it can reduce the number of times to apply for memory space for intermediate data during the iteration,thereby it can reduce GC time.By optimizing the calculation method of the gradient matrix in the TV algorithm and merging the calculation of the partial matrixes,it can reduce the computation amount of the algorithm.The accuracy and acceleration of this method can be verified by comparing with the single serial experiment.