Data Quality Analysis And Optimization In Public Security Intelligence Based On ETL

With the continuous development of the public security sector, this public security system has mastered increasing types of data on the Digital City, Golden Shield, and Peaceful City projects, amongst also numerous other projects. Additionally, the system is becoming more and more complex. However, the ongoing problems stemming from public security intelligence information assessment, holds no sufficient conformity of the resources, the limits of information sharing, low level analysis; all of which cannot form an effective mechanism to integrate various types of data resources and comprehensive analysis.The development research of the system logical model bases on ETL, are the hot topic and focal points of data warehouse construction, both at home and abroad. In this thesis, the ETL technology is applied to the intelligence analysis system, the structured data is extracted, cleansed, converted, then a variety of data types are integrated, creating unified data standards which are then used by comprehensive analysis of intelligence. However, data quality directly affects the credibility of the final judgments, and the direction of decision making; for this reason, data quality of is very important to the reliability of the final output.This thesis, is a process of data quality testing and optimization, designed and implemented with a particular formula of data processing producing quality data in the public security sector.The primary works in this thesis are presented as follows:(1) using ETL tools to cleans and convert the data, the different types of data are synchronized to the system's data warehouse from the source data system.(2) the rules of data quality inspection are defined from the data integrity, timeliness, business normative, which are utilized to detect and estimate the data quality.(3) the data quality optimization system is designed to analyze the problem data, complete and revise the data. The data source system is applied to complete the data according to the cause of the problem, meanwhile improving the ETL conversion process, so the data quality is gradually and continuously upgrading.(4) The levels strategy of quality inspection is designed. The different levels of testing tasks gets different execution cycles, which is distinguished on the basis of importance of level of task, the detection task of data is designed to JOB as well, which is implemented based on needs.(5) The data conversion effects of the platform is tested on the functionality and the performance, that is to test validity of the data and function after data conversion.This thesis combines theory with practice, studies data integration and sharing and quality control, sets up a data quality evaluation system for data storage layers, data collection layers, index layers, and rule layers to improve assessment of the problematic data combined. This thesis strives to create accurate data, and accurate judgments.