The Study And Implementation Of Distributed Knowledge Search System

With huge amount of valuable information on the web, search engine becomes an important tool for information retrieve. Traditional search engine could not understand the syntactic meaning of web text, and provide web page link list based on key term match and PageRank value. With the increase of web data and user’s need for accurate search, traditional search engine gradually can’t satisfy use’s demand. To overcome the short point of traditional search engine, knowledge search engine is invented. Knowledge search engine analyze user’s query and provide user with knowledge in form of entity and entity relationships. In consider of the high time consuming of NLP and the safety of knowledge storage, we think it’s a good choice to make a combination of the knowledge system and a distributed framework. This paper implements a distributed knowledge search system consist of a workflow framework, a distributed network crawler and a modular for distributed knowledge extraction, we can change the workflow of this system by configuration. After all, this paper made a contrast of the single knowledge search system and distributed knowledge system which base-on a3-nodes Hadoop cluster. The experiment shows that the distributed system has a double efficiency then the single node system, and the efficiency can be improved by enlarge the cluster. Also, the distributed system guarantees a better performance of storage safety.