Research And Implementation Of Data Service Platform In Smart City

In the process of globalization and urbanization,the smart city has become an important strategy of our country.The data service platform is an important part of the urban construction of a smart city,and it stores the basic data of the city's urban management,moreover,it acts as the information and resources service platform for decision-makers to carry out the urban management,command and dispatch.It is the key factor for the city's operation and management,any unavailability of hardware or software is a disaster.At present,all the functional departments in the region not only stored large amounts of data resources,but also the data formats types complex even diverse.Therefore,In order to classify,manage and exchange data in a structured way,the data service platform must set up a reasonable data storage and management scheme based on the structure and characteristics of the existing data in each business system,investigate a set of highly available and extensible platform architectures to ensure that platform data services are continuously available.The security of the key data and the security of the platform itself is one of the foundations for ensuring the effective operation and management of the functional departments in the region.Therefore,the data service platform must also strengthen the prevention of data security.In current research,the data model is constructed by reverse engineering technology.Based on the deep analysis of the data model,a reasonable and unified metadata standard is established for all the data service platform users.The data service platform adopts the storage method originated from My SQL Cluster and distributed file system HDFS,according to the metadata specification to strictly classify and store the data as its type and scope.The platform uses Zookeeper to provide distributed coordination services for the entire Hadoop cluster,uses Redis to be the cache between the storage layer and the application layer.To mine the great value of the data,the data services platform employs Hive and Spark to provide more efficient offline computing and online real-time analysis capabilities,uses YARN to manage and control the resources in the cluster,and finally exploits Zabbix to monitor the nodes and services in the whole online environment.In order to improve the usability and expansibility of the data service platform,present thesis uses HAProxy,Keepalived,Mycat and My SQL loss-less semi-synchronous replication technology to guarantee the high availability of My SQL cluster,adopts shared storage system and Zookeeper to realize the high availability of Hadoop cluster.The hierarchical model of the data service platform is designed based on the main idea of SOA architecture.Services of each layers interact manually through a well-designed interface,provide a unified access interface for all data resources services through the RESTful API.Follow the principles of security,this thesis uses firewalls,intrusion detection equipment,gatekeepers,security audits,private line and other network security methods to strengthen the network control.Securing control of Hadoop clusters in a way that uses fine-grained service authorization and integrated third-party authentication system Kerberos.From the technical and regulatory aspects,the unified certification center provides identity authentication and resource-based access authorization services,to makes the core data resources safer in the data service platform.Via the research and verification of the storage and management scheme,the high availability scalable platform architecture and the security scheme of the data service platform,the results confirm the rationality and validity of all the work schemes in this thesis.The data service platform is able to perform accurate classification and storage of all kinds of complicated data,and can be used in a flexible way,breaks the fragmented data island status of existing system,promotes the effective integration of data resources,provides more efficient,reliable,security service for the region's functional departments to carry out the daily business and the construction of better urban service for people.