Design And Implementation Of Data Monitoring System Based On Log File

With the increasing development of Internet technology, the network has provided people with vast amounts of information and diversity of entertainment environment, such as news, television video, music, etc. However, faced with various of information and services, how to help the users to conveniently obtain to the information they really want and interested in resources.So personalized recommendation engine technology arises at the historic moment.With personalized recommendations, user’s browsing and retrieval costs can be reduced. The core of personalized recommendation effect is recommended strategy and algorithm. But due to different preferences of users, complexity of the demand, and diversity of informations, offtering effective strategies and algorithms cannot be done at one stoke, According to these difficulties, analyzing and evaluating relied on the recommend ation engine log, repeat iterativing, continuous improving and optimizing are really necessary in the experiment and the process of online use. However, faced with thousands of products experiments, and massive user access logs, how to monitor real-time and effective control of the use of the products, so as to assess the effect and promote the recommend strategies and algorithms’ optimization and improvement, which has become a very interesting and chanllege topic.This paper designs and implements a massive log data monitoring system based on recommendation engine log. The monitoring system mainly for product testing and massive user access log to monitor and analyze, and according to the results in rapid objective assessment the effect and quality of the recommended products, and to the product or the promotion effect of quantitative evaluation.System gets log data through the filter processing from a data warehouse based on HDFS, and then using the temporary framework completes the statistic of customization experimental index and imports the statistical value into database.Finally,the results by monitoring data in the My SQL database in accordance with the configured monitoring strategy, will be send to the appropriate person in charge of experiments and products through e-mail and text messages. Provide data to support real-time optimization for the recommended impact assessment and monitoring strategy and algorithm.