Research And Application Of Distributed Index In Donghua Search Engine

With the rapid development of IT technology, social life has been entered into the era of information explosion. The resource in Internet is increasingly rich and the application scope is growing, search engine play a very important role, for access to information provides great convenience.But the growth of users and data to the traditional centralized search engine is so overwhelmed, thus derived, is the search engine performance issues.Through the detailed analysis and discussion of relevant theories and techniques of search engine, this article will focus on the search engine index, research and develop the framework of scalable distributed index, and eventually used in search engine of Donghua University.Main research contents are as follows:Discusse the current status and trends of the development of search engines at home and abroad; analyse search engine works as well as the index part of the organization and processes; research and analyse distributed technology theory, load balancing theory and Map/Reduce calculation model; Pairs of open-source Lucene search engine tool kit and RMI (Remote Method Invocation) technology, analyzed and studied.Main contributions are as follows:1 .Using the Lucene core package to create inverted index, based on RMI the documents that will be indexed will be hashed into distributed enviroment,creating process distributed computing,index results distributed storage.2.Reference Map/Reduce model to establish parallel processing .query calculation decompose, parallel computing,then merge.3.Through the load balance, the balance between the nodes of the load-stress calculation, reasonable dispatch distributed resources and complete data retrieval.