Design And Implementation Of Extensible Digital Signal Processing Algorithm Library Based On Spark

With the fast growth of Internet technology,the intelligent manufacturing industry has received extensive attention.Amount of analog signals accumulated by industrial equipment has increased explosively.The traditional digital signal processing technology based on DSP and FPGA is more suitable for real-time signal processing.And due to the limitation of data size and frequency resolution,it is not suitable for off-line data processing,analysis and mining under large scale data.Spark is a parallel computing engine for large-scale data processing.It has wide applications in big data processing.This thesis relies on the industrial large data analysis platform.With the advantage of parallel computing of Spark,Spark is used as the computing engine to accelerate real-time signal data processing and off-line signal processing.However,the platform has some shortcomings in function.For example,the analysis platform is short of mathematical computing solutions,such as digital signal processing,which is suitable for distributed parallel computing engine.This thesis proposes a distributed digital signal processing algorithm library based on Spark,which provides support for the analysis oriented industrial big data application scenario.This thesis introduces the structure,architecture design idea and implementation details of the algorithm library in detail.And with the fast Fourier transform(FFT)algorithm,the fast realization scheme of the discrete cosine transform algorithm(B.G.Lee algorithm)and the fast realization method of the wavelet transform algorithm(Mallat algorithm),the idea of distributed realization of the traditional digital signal processing algorithm in Spark is introduced.Finally,the correctness of the above three distributed algorithm is tested,and then the correctness and performance analysis of the algorithm library is carried out.The results show that the algorithm can complete the function of digital signal processing correctly and can meet the demand of digital signal processing for large-scale data set by industrial large data analysis platform.