The Research And Application Of Data Integration System Based On University Big Data Platform

With the construction of digital campus,data query and load efficiency of the traditional data integration system in the massive data environment are reduced.It is difficult to integrate unstructured,semi-structured data fusion and analysis in the massive data environment.For the above,relying on university large data platform,combining the advantages of Hadoop and MPP technology,a system of heterogeneous data integration based on MPP-Hadoop hybrid framework is designed and implemented,which integrates many different structure data and enhances the efficiency of data query and loading.And taking a university as an example,the students trajectory data is extracted from the student's access card system and the campus network system and is loaded to MPP data warehouse.The system will be compared with the traditional university data integration System built by Oracle data warehouse.The validity of the system is verified by the comparing result.Technical support and guidance to students' life,study,psychology and other aspects of management is provided.The system will be compared with the traditional university data integration System built by Oracle data warehouse.The validity of the system is verified by the comparing result.Technical support and guidance to students' life,study,psychology and other aspects of management is provided.The main research work of the thesis includes:(1)The thesis introduce the background and significance of the issue and introduce the research current situation at home and abroad.And expound and compare the main data integration technology,for example,Federated Database System technology,middleware technology,data copy technology.(2)Relying on university large data platform,combining the advantages of Hadoop and MPP technology,a system of heterogeneous data integration based on MPP-Hadoop hybrid framework is designed and implemented,which integrates many different structure data and enhances the efficiency of data query and loading.(3)Expounding the key module of the system.(4)Analysis the students' behavior trajectory can prove the validity of the system.