Research Of Urban Hotspot Traffic Analysis And System Realization Based On Spark

In recent years,mobile internet develops rapidly and has become an indispensable part of people's lives.Base station as the hub of user and Internet connection,the quality of network service provided by base station directly affects the user's experience.How to evaluate the quality of network service in different areas and at different times in the city and identify the hot base stations in the city are the issues that operators need to pay attention to.The analysis of the spatial and temporal characteristics of base station traffic,number of connections and types of Web site visits provides guidance for operators in improving the quality of network services,improving the utilization of network resources and planning of future network facilities.Based on the traffic data of mobile base stations,this thesis proposes a definition method of hot base stations based on entropy weight method.On this basis,the time distribution and spatial distribution of hot base stations are studied.In this thesis,naive Bayesian algorithm is used to classify web sites in mobile Internet,and TF-IDF algorithm is used to explore the importance of different types of Web site access to hot base stations.The study of population movement and population distribution is of great significance in urban planning and traffic construction.In this thesis,firstly,thiree models based on time series are used to predict the number of base station connections,and the performance of the three models is compared.Then,a method based on spatial-temporal characteristics of base station is proposed.Experiments show that the performance of the proposed algorithm in most base station data sets is better than that of only using time series.In the big data environment,the traditional data processing tools are no longer applicable.This thesis uses Hadoop to complete data processing.However,the open source big data processing tools have some shortcomings in security,authority control and result display.Based on this problem,this thesis develops a Spark-based traffic analysis system,which integrates three functions,i.e.,Spark Shell,Spark SQL and Spark job submission on the web side.This system provides convenience for large data analysis including but not limited to traffic analysis.