Research And Application Of Distributed Demantic Neighbor Search Algorithm Based On Spark

On the Internet,a large amount of high-dimensional data is generated in an exponential manner.At present,the research and application of such data are subject to dimensionality reduction,clustering,and neighborhood search.For these massive amounts of data,it is very meaningful to improve the accuracy and speed of semantic search.This paper mainly studies the affinity propagation clustering algorithm based on firefly algorithm and the distributed implementation on Spark platform.Based on this,the development and application of semantic search engine in technology field is carried out.The main research work of this paper is as follows:(1)Study a affinity propagation clustering algorithm based on firefly algorithm.The optimization of affinity propagation clustering algorithm can not adaptively adjust the bias parameters and the clustering effect is not good.The firefly algorithm is used to dynamically detect the optimal cluster center,instead of the original algorithm to default each data point to the cluster center point.(2)Based on(1),the research of distributed firefly neighbor propagation clustering algorithm and its implementation on Spark framework.The massive high-dimensional data set is divided into several subsets.Each subset is selected by the improved affinity propagation clustering algorithm to select the cluster center points,and then the cluster centers of all subsets are merged again to cluster.The distributed algorithm is designed and implemented on the Spark framework.(3)Study a semantic search engine in the field of science and technology.Design multi-level semantic index structure,use(2)algorithm to cluster technology resource semantic vector,build multi-layer semantic index,take the union of multiple clustering results in index when semantic search,speed up semantic search and improve accuracy degree.Based on the above research results,the big data semantic search engine in the field of science and technology was developed and applied to the “Saku Chuangzhi”big data technology achievement transformation service platform,which effectively promoted the precise matching and docking transformation of scientific and technological achievements and local needs.