Search Engine Based On Distributed Storage

Distributed storage is a good solution to solve the problem of data storage and peripheral components application under the scene of search engine.Traditional distributed storage is mainly based on disk storage and only solves the distributed problem of storage.The access delay of the whole storage is not controllable.Traditional distributed storage has a certain labor cost of operation and maintenance,failure node cannot be automatically restored,and the complexity of operation and maintenance is relatively high.By using skip table,the multilayer sparse index is realized,and the basic data structure of memory index is realized.By taking Etcd as the registry,the design of Dispatcher service,together with distributed nodes,solves the fault-tolerant problem in distributed scenarios.Cluster management is realized based on Kubernetes.Data flow and node fault recovery process are fully automated through the cooperation and combination of Dispatcher service and cluster management service,without manual processing,which reduces the operation and maintenance labor cost and operation complexity.By combining index service,Dispatcher service and cluster management service,a new distributed storage form is formed.The new distributed storage service can well control the overall access delay,and can dynamically perceive the node changes,without being uncontrollable due to traffic fluctuations and node anomalies,and reduce the operation and maintenance costs to the greatest extent.By invoking the distributed storage service through the upstream service,it can be well applied to the search engine scenario and solve the distributed storage system requirements of the search engine scenario.