Research And Application Of Data Integration Processing Techniques Suitable For Statistic Analysis

With the rapid development of China's urban public transport, rail traffic that has been gaining in favor is an important component of traffic in the city. At present, however, AFC of rail traffic remains in automatic monitoring operation of the device, ticket information transaction and management in the application of computer technology level. In this paper, the project studies for mining, discovery, statistical analysis, utilizing the massive data of ticket transaction, equipment condition and maintenance log. This paper focuses on object integrating technology and applications suitable for the statistical data.The main research and innovation of the author are as follows:i) In order to facilitate the users to understand and use various data sources organizational structure information, propose a method that enhance the ease of use of the system. In this method, it maps database metadata information and semantic description of Chinese, and English Dedicated Short Term described as easy-to-understand Chinese semantic description;ii) Through research based on the Linux platform data integration, using XML technology to solve the problem of heterogeneous database data conversion, storage and data relationship operation. According to the characteristics of the actual operational data, research and realization of the data conversion include: format amendment, field decoding, the units of measurement conversion and date/time transformation, and in conjunction with the relevant data warehouse strategy and the structure of the project, the paper propose a two-tier data conversion model. The model in realization of the development has a greater flexibility, which is taking up less computing resources and having strong expansion in the deployment and operation;iii) With the concurrent response for request and data file storage, through analysis characteristics of document storage and XML format file storage, the author defines a XML storage format file which is suitable for statistical results. It is a good way to meet the application needs of the results cache and re-feedback, and can control storage space strategy of outcome document in XML format;iv) For data calculation, the paper proposes a strategy that makes some partial relationship operation and numerical computing strip from database system. This strategy can reduce resources occupation of the business database system. Data files based on XML can be read according the XML DOM interface, do relevant relationship operation, numerical calculation and other operations. In order to improve the process efficiency of connectivity, the author achieved the NES-JOIN algorithm based on the XML data files.Proven practical applications, this paper deal with the data integration technologies and methods can meet the online analytical processing application needs, and optimize the use of production site host computing resources.