Research And Implementation Of WIFI Fencing Big Data Processing Mechanism

WIFI Fence Collection and Analysis System is based on big data processing system,it is committed to finding the available information from the user,through the user to connect WIFI,hot spots and other wireless network information analysis to determine their appearance Location,time and other information,and then draw the user's trajectory,which is of great significance for people travel or government public security organs to find the suspect trajectory.The main research and work content of this paper are as follows:1.Introduced and analyzed the main big data distributed storage and calculation and analysis technology.2.Studied the key problems of this paper,and studied from four aspects:large amount of small files in big data storage,HBase massive data import,HBase query optimization and HBase query paging solutions.3.The system needs analysis,analysis of the system data sources and the size of the data,and then analyze the system with big data related to the specific functions and non-functional requirements.4.The overall architecture of the system design,respectively,given the physical and logical architecture,and then the big data processing module design and implementation of three,namely data storage module,data analysis module and data query module.This system uses Hadoop HDFS to store the collected data.In combination with the large number of small files stored in the research section,a file merging mechanism is designed to improve the data storage and improve the data loading performance.At the same time this system uses Spark as the calculation model of the system architecture.By studying SparkSQL and Spark RDD design a more efficient way of handling data.This system designs a kind of HBase data import mode based on Spark,and carries on the performance comparison with the traditional BulkLoad batch import way.The system also uses the HBase distributed database for fast real-time query,HBase query optimization research on the system HBase query optimization,design of HBase query paging based on Redis,and improve the system query efficiency.Finally,a distributed cluster of the system is set up,and the performance testing of each module of the system big data processing is conducted.The performance of the proposed key problems is also analyzed.