The Design And Optimization Of Distributed Search Engine's Cache

The internet has huge information resources and it's a big task to search and pick up useful information. Search Engine just provides the service of universal information searching.A search engine is usually composed of Crawler, Indexer, Searcher and User Interfaces. Crawler is used to download web pages from the internet, parsers to parse them in order to make index, Indexer to make them a better way for searching and save them in Indexing Database, Searcher to match the query keyword and destination document, User interfaces to provide a web page for user to input query and return the search results.As the data Search Engine processes is so huge and the internet is distributed, it's better to make Search Engine a distributed system with multiple computers processing together, which will provides better performance.The paper gives us a discussion of a web Search Engine structure based on distributed parallel computing technology. It also used distributed parallel programming mode, task delivering mode and relevant decomposing technology. In parallel programming, thread is a popular mode, while C++ Thread Pool is a goad one in parallel computing. For distributed parallel programming, message delivering is effective and we used the socket communication in C++.In the design of web search engine, we mainly show you a Buffer Optimization method, which is the key to the speed of Search Engine and its performance.