The Research And Design Of Distributed Search Engine Cluster Based On Lucene

With the continuous improvement of social informationization, the traditionalcentralized information retrieval technology based on single system has been unable tomeet the demand of concurrent multi-user parallel information retrieval based on largescale data set. Using the high speed network environment to build a distributed searchengine cluster system for distributed information retrieval, now has become a new trendof development of search engine.Inflation of information set increase system maintenance cost and search responsetime. In order to adapt to increasingly high requirements for modern retrievalenvironment, structure design and algorithm optimization of retrieval system is still ansignificant research direction.This paper, about indexing and retrieval that the information retrieval systeminvolved mainly in two major areas, respectively, puts forward two kinds ofoptimization algorithm, and derives several different technologies to meet the needs ofparallel and distributed applications. In the index, because of the single RAM indexingand the single FSD indexing are existing many flaws, this paper insists RAM-FSDcollaborative indexing, and derives RAM-FSD collaborative parallel indexing andRAM-FSD collaborative distributed parallel indexing technology. In the retrieval, inview of the existing thread pool disadvantages, this paper puts forward a kind of newthread pool implementation method. On this foundation, this paper brings forwardthread pool optimization retrieval technology, and derives a parallel retrieval thread pooloptimization technology and distributed parallel retrieval thread pool optimizationtechnology.In the design of distributed search engine cluster, this paper discusses problems ofgenerally distributed search engine system, absorbs the advantages from GFS,thenshows a distributed cluster system with the characteristics of security, sufficient, easyexpansion, shared resource and low cost.