The Design And Implementation Of Data Processing Service In Monitoring And Disgnostic System Of Index Distribution

Distributed software system consists of multiple software module distributed on different computers through network connection,has great advatanges in resource sharing,high availability and parallel processing.As a consequence of the distribution in resource,complex interaction and tremendous scale,traditional test methods fail to ensure the correctness and reliability of large scale distributed system(LSDS),unexpected errors occur constantly during system runtime.Since the error detection and diagnosis of LSDS has became a big challenge in reality,monitoring is regarded as a commonly used approach in the ensurance of reliability of LSDS.The background of this thesis comes from my internship during eBay search department,index distribution team.The ultimate goal of index distribution is to ensure that more than 20000 query nodes get the correct and latest index files,which is a complex process performed by mutual interaction of different distributed systems.Due to the lack of complete monitoring,diagnosis and remediation mechaisms,location of problem root cause often takes a long time,which in further decrease the user search experience.A monitoring and disgnostic system for index distribution was proposed in this thesis,aiming to solve the aforementioned problems.Project background was summarized,the research progress on both distributed index technology and software monitoring are overviewed.The design and implementation of data processing service(DPS)of the proposed system are focused in this thesis.Akka Cluster is used to build cluser service.Based on the distribution and execution of rule file,the health of index distribution system components was monitored.Quartz is used in the scheduling of rule task.The monitored objects and system code was separated,the flexibility of monitoring service was improved.The diagnostic workflow framework was implemented.Combined with the remedation service,DPS can determine the specific root cause of system failure and make corresponding system remedies.DPS can automatically detect live site issues,analyze root cause,send report,do the remediation and open to onboard new use case easily with scalability and stability.DPS shortens the system fault detection and repair time,reduce the need for human intervention.Developers can focus on the business function development.The robustness of index distribution service was improved,users can get more accurate search results.