Design And Implementation Of Population Data Analysis Platform Based On Big Data Technology

The analysis of population data is an important foundation for scientific decision-making,and population data is the basis and premise for making any policy.The current population statistics cannot meet the needs of population management and the need of improving the quality and level of public service.The existing city population database has encountered bottlenecks,and it has incomplete data fields and lack of effective verification mechanism.With the passage of time and the development of the business and the arrival of the big data era,the amount of data and frequency needed to be exchanged will increase and are unable to meet the growing business needs.Therefore,it is necessary to establish a set of population data analysis platform through big data storage,calculation,analysis and mining to solve the above problems.It is a huge systematic project to record the attributes,characteristics and changing rules of population in real time.Nowadays,the population data has become the big data.The construction of population data analysis platform requires the use of big data technology.The population data analysis platform aims to use big data technology to achieve the centralized definition,centralized storage and centralized use of all kinds of basic data,management data and operation data.For updating more frequent data,we use Spark technology to do fast calculation and analysis.For updating data that is not too frequent,we use Hadoop technology to do data calculation and analysis.Besides,Hadoop cluster itself uses Zookeeper technology to do job coordination and scheduling.The function includes,using big data technology,in accordance with the implementation of various kinds of data collection,preparation,modeling,the results,demonstration,archive storage,grading to achieve value-added data storage and management framework application environment and technical support that above functions needed.The platform provides the global uniform data comprehensive utilization and the management environment for upper application system.It also provides uniform data storage,load,query,analysis and display platform framework for all kinds of specific application(application subsystem or a module).