Research And Implementation Of A Distributed Web Services Search Engine Based On Map/Reduce

In Web service technology, UDDI, as the platform of Web service publishing,discovery and communication, services as a bridge between users and Web servicepublishers, with the closedown of public UDDI one after another, more and moreservice providers tend to use the non-registered way and publish their services in theirown Web sites directly.. So the deployment of the Web service shows a moredecentralized trend, causing the problem that it is much more difficult for users tosearch and find Web services interested, under this situation, study on the use ofsearch engine technology to find Web service starts to become a hot issue recently. Ithas been more than ten years since the concept of Web services was first mentioned.Along with the wide application of Web services, it also exposes more and moreproblems. At the same time, since Restful Web services can solve the defect of thetraditional Web service effectively while at the same time inheriting the advantages, itattracts more and more attention of enterprises and researchers. Web API is a typicalRestful Web service. At present, more and more Web APIs are opened, but it is stillvery scattered in relation to the Internet, a very practical research problem is how tocollect the released Web API as much as possible for developers to use. Map/Reduceis a distributed programming model, it designed and implemented a algorithm that cancompute and store large scale data set under the environment of multi-nodecalculation, By the use of Map/Reduce technology, it is convenient and efficient todeploy distributed application in ordinary PC cluster, and it can effectively improvethe performance and efficiency of the search engine system.The major work and contributions are as follows:1. A distributed Web service search engine architecture based on the Map/Reduceis proposed, and the related modules are introduced in detail.2. By using Map/Reduce technology, the entire system is constructed in theHadoop frame, it gives a full play in Powerful computing and data storage capacity ofthe computer cluster, experimental results show the efficiency of the system ingrasping Web service, it performs much better than the traditional centralized Webservice search engine.3. An improved URL management mechanism is proposed through the researchof distribution situation of Web services on the Internet, the experiment proves that the efficiency of the search engine can be improved by improving the URLmanagement mechanism.4. A vertical and analysis search algorithm for Web API description document isproposed by analysis and research on the accessed Web API description document, theexperiments show that it is feasible to use the Web spiders technology to search andgrab the Web API describing documents on the Internet.