Design And Implementation Of News Log Analysis System Based On Big Data

In the information explosion era of big data,who has the data has the initiative,so the data has become more and more important.The log data online as a user behavior operation records,on behalf of the user's behavior preferences and user habit for applications,so a growing number of Internet companies,want to use the user log behavior data,to unearth more valuable commercial information,therefore,log analysis system has become more and more important.This paper firstly discusses the background and significance of news log analysis system,and introduces the existing log analysis systems,then,summarizes the problems encountered in the practical application process of news log analysis system.After that,this paper deeply studies the big data processing framework Hadoop and Spark used in the current mainstream log analysis system,and introduces the technical key points of the log analysis system,such as distributed message queue,log acquisition framework,distributed coordination service principle and application scenarios.Next,in order to solve the problems existing in log analysis system.This paper designs a system from the user management,data collection,data analysis,system monitoring and data showing these five functions,and applies a scheme that combines big data processing technology,distributed message queuing and distributed consistency.Finally,the realization of the news log analysis system greatly improves the efficiency of data analysis of business personnel and plays a good supporting role in the analysis of news business.