The Design And Implementation Of Real-Time Monitoring System Of Search Engine Based On Akka

The arrival of data times brings fast improvement to data quantity.It's no doubt that technologies related to big data provides our life with great value.In the area of e-commerce,by analysing huge amounts of search data,we can predict users' demands more correctly and recommend suitable goods for users,which improve user experiences of search engine.Due to the velocity feature of search data,we needs to process it within a limited time.This factor raises higher demands on the processing ability and architecture model of software system.As an e-commerce company,eBay needs to process search requests from hundreds of millions of eBay users all over the planet every day.Its search engine will generate a huge quantity of data.eBay's current monitoring system of search engine is based on the previous search engine Voyager,it's not able to satisfy the basic monitor requires because of the increasingly large quantity of search data.This deficiency has become particularly evident after eBay launched its new search engine Cassini.lt's a task full of impendency and reality for us to develop a real-time monitoring system to satisfy new features of Cassini.This real-time monitoring system of search engine is generated in such background.The system collects data from eBay query node cluster,schedules real-time jobs to monitor data.When detecting abnormal data,it sends alerting mails.Meanwhile,the system supports the display of monitoring data.The system uses Scala as developing language,adopts distributed architecture,uses Akka as real-time stream processing framework and uses Kafka as message system.The system is devided into four modules:data collecting module:it collects data form query node cluster;monitor scheduler module:it schedules jobs of monitoring and alerting;data interacting module:it provides an interface to aceess the database;data exhibition module:it demonstrates the monitoring data as a graph or chart.This system has successfully detected abnormal data of search engine and provided visual demonstrate of search data.It has effectively improved search experience of search engine after launched,which meets the requirement of search engine monitoring.