| With the rapid development of the Internet,the company's business continues to expand,the corresponding business data,log data also grew rapidly.With the continuous growth of business data volume,traditional database becomes increasingly inadequate for multi-dimensional real-time query and statistics of large-scale data.With the rapid growth of log data,the problem of inefficient storage and acquisition will arise.Large-scale log data cannot be retrieved and analyzed centrally,and it is difficult to obtain valuable information from it.Based on the problems encountered by the above enterprises,it is necessary to select an open source technology and design a scheme to apply it to the business scenarios of enterprises to meet the requirements of enterprises for multi-dimensional comprehensive real-time query statistics and data storage of business big data and log big data.This paper introduces the application of search engine ES in enterprise big data by taking the reconstruction of second-hand car industry-auction list service and the establishment of log retrieval platform as examples.The main research work includes:1)combine the research background of the subject and the actual business requirements of the enterprise,and organically combine the search engine and Java web software system.2)based on various search engine technologies in the market,the technology selection of distributed search engine is made by integrating the company's technical reserves,the fit of scene application,product stability,the activity of open source community and other factors.3)deploy ElasticSearch service instance based on its technical characteristics,implementation principle,installation configuration and extensibility design.4)based on the business requirements of the auction system of the second-hand car industry,redesign the auction system vehicle service list and reconstruct the auction vehicle list service.The test results of the modified service are analyzed statistically.5)integrate the Kibana visualized analysis platform and Logstash log collection with search engine based on ElasticSearch to set up the log retrieval analysis platform.The log retrieval platform is used to count the customer access log and analyze the user behavior.The research results of this paper show that the introduction of search engine into the business software system can improve the performance of the service,improve the stability of the service and improve the user experience of the service.The log retrieval platform built by search engine can solve the problems of large workload of log search,non-visualization of log statistics and non-dynamic expansion of log data storage.The research results of this paper also provide some reference value and technical basis for enterprises to use search engine in more business scenarios.The statistical methods and analytical methods in this paper can provide some reference for enterprises in big data analysis. |
