Research And Development Of University Search Engine Based On Scalable Distributed Architecture

In recent years, the information in college information system has a growing trend and exploded increasingly with the rapid development of Internet technology. The resource in it is increasingly rich and the application scope is growing. By the crawler detection, the amount of web pages which can be visited on the web site of Donghua University was reached more than 100000. The university's website is not the only demand of teachers and students. The model course information, admissions information can also be integrated into the needs of them. Their needs enhanced the dependence of them to search engine technology.The research in this paper is based on the information retrieve demand of DongHua University, studied the basic search engine architecture and scalable distributed architecture in consideration of the increasingly information amount and visit load. In this paper, it describes a scalable distributed architecture based on linux system and open source software. It provides capability to expand the system performance without interrupting the online service. The works and research results are as below:1. Studied the theory of search engine and information retrieve, such as the algorithm and architecture of web crawler, inverted index and search result sorting.2. Studied the theory of distributed system, such as load balance system, distributed cache, distributed index and Map/Reduce model.3. Designed the basic architecture of Donghua University search engine which consists of crawler system, preprocess system and search system.4. Designed the scalable distributed architecture of Donghua University search engine which consists of LVS cluster, memcache distributed cache and distributed index generated by Map/Reduce model.5. Developed and implemented Donghua University search engine according to the above structure and evaluated the performance and precision of it by the experiment.