The Design And Implementation Of Distributed Search Engine Based On MPI

Nowadays, we are in an era of information. The explosive growth of Web data following the development of the Internet and Information Technology. Because of the large amount of information, people are mor difficult to find what they want from Internet. Just under such a background, the search engine is more and more inportant in people’s life.The network resource on CERNET (China Education and Research Network) is very valuable for the user in college. However, there is a defect that the CERNET always be ignored by the mainstream search engine. The goal of this article is to design and implementation of a new distributed search engine, and provide web page retrieval service for teachers and students in university.Basing on the deeply researching of the search engine technology, this article design a distributed search engine base on MPI (Message Passing Interface) and implementation some important modules. The distributed search engine system uses a distributed architecture, and has the good extendibility and the modifiability. In the implementation of the system, this article design implements an algorithm computing the PageRank in distributed system. And design implements a subsystem computing the inverted index using the MapReduce model.