A Distributed Log-search System Based On Elastic Search

With the development of cloud storage and cloud computing technology, mass data’s storage, index, query has become a crucial technological bottleneck for the Internet data management centre. Because the current internet data are massive, the relational database base on the ACID theory is now hard to meet the needs of current mass data management. According to the theory of CAP, it is impossible for a distributed system to meet the needs of consistency, availability and partition tolerance at the same time. A relational database which can ensure consistency and availability would be difficult to carry out the partition for its structure and deployment either horizontally or vertically. Thus it is deprived of the potential scalability which may achieve by utilizing parallel system or distributed system.The thesis puts forward a carefully worked-out plan for the framework and design of a distributed log-search system as well as its deployment strategies based on Elastic Search after a thorough study of the design principles, application environment and real cases of application of the current search engines. One the one hand, most of the queries base on the primary key, on the other hand, the system doesn’t need to ensure strong consistency. So, in order to achieve stable, reliable, friendly real-time search demands, the system uses distributed strategy to storage, index and cache data.As this plan takes the characteristics of mass log-data into account, the system performs well on data insertion and searching and is able to ensure the availability and eventual consistency at the same time according to the test results. Thus this plan can well reduce the costs of deployment and operation. The distributed log-search system put forward here is capable of meeting the different needs of mass log-data management such as data storage, parallel query and classification statistics while keeping its efficiency at a high standard.