Design And Implementation On Distributed Product Serach Engine Based On Hadoop

In the field of electronic business,a variety of online shopping platform developed rapidly in recent years,and online shopping has become a basic method a lot of people use to buy goods. Meanwhile, with the popularity of C2 C business model, more and more people set up shops online.Merchants can open store and update item information at any time they want,along with the vast amounts of product information updates.How to update and gather these data timely and accuratly, how to let consumers quickly find their own favorite goods at online shopping platform, the product search engine of online shopping platform are facing enormous challenges.At present,most of the search engine system are structured to centralized structure, which means all of systems’ modules are deployed on one server, and it also result in the server must be of high performance,meanwhile,the system still have poor stability and bad scalability. In order to deal with these disadvantanges, people have to purchase very large and expensive servers to satisfy the system requirements.Here we come up with a distributed product search engine in vertical field,except the data crawler,this search engine contains functions of building index,query search,cluster management,service management,monitoring,etc.This distributed engine is provided for inshop search,which was developed by multiple team members.Compared to many existing engines,this search engine can provide the same magnitude in consumption(billion items) with fewer resources and faster retrieval speed.For some large-amount and high-frequency query,we come up with a design of truncated index,which solve the problem of slow search by long index list.A t the same time the engine has good stability and could support a variety of search services,the design of key-value format field in original documents makes it serve for not only e-commerce sites but traditional websites.