Design And Implementation Of University Data Sharing And Exchange Platform Based On Data Middle Platform

In recent years,colleges and universities have been endowed with a deeper mission of education,which requires the information work of colleges and universities to follow times step closely,strengthen research and innovation,carry out work with more wisdom and scientific technology and tools,and respond to the increasingly diverse needs of campus teachers and students with more refined services.With the continuous development of university informatization,more and more work is completed by the information system,resulting in the rapid expansion of information resources and various types,the lack of integrated use of data,poor exchange and sharing ability,teachers and students face with massive data but cannot get effective data services.In the era of big data,how to solve the problem of data management,turn the huge and jumbled data on campus into usable resources to generate value continuously,and provide high-quality services for the teaching process is the topic of urgent research at present.With the popularization of the concept of data middle platform and its advantages in data fusion,processing and analysis,it provides a direction for data governance and data sharing and exchange in universities.According to the current data of Beijing Jiaotong university and the real demand analysis,this thesis innovatively proposes a comprehensive solution of data exchange and sharing platform based on data middle platform,including designing and implementing the data collection and circulation subsystem,data governance subsystem,data development and analysis subsystem and data assets service subsystem,helping to build an efficient and sustainable data asset system,give full play to the value of data and effectively manage it.The data flow and acquisition subsystem involves the design and construction of the real-time data warehouse,which collects the source data dynamically and stores it into the data warehouse through the data extraction tools and technologies of the integration of flow and batch.The data governance subsystem is mainly responsible for the overall management and maintenance of data standards,metadata,master data,data security and data governance in the data middle platform.Data development and analysis subsystem mainly supports system users to manage data development tasks and provide data kinship and multidimensional data analysis functions.Data asset service subsystem mainly provides API interface,data catalog,data query,data analysis,intelligent push and other services to realize the value of data asset.In the process of platform development,the author was mainly responsible for the design of the architecture of the whole data middle platform,the construction of realtime data warehouse,and the design and implementation of the four subsystems of data acquisition,data governance,data development and analysis and data asset service.The system is developed based on the Spring Boot framework,mainly using Flinkx,Canal,Kettle and Flume tools for data collection,and using My SQL,Redis,Hbase,HDFS,Click House and Kafka as data storage.Flink,Spark,Atlas and the sdk of IFLYTEK were used for data development and analysis,and Davinci and ECharts were used for data visualization.The project to realize data exchange sharing platform has been put into test run in Beijing Jiaotong university,which greatly helps the university alleviate the current difficult problems such as inefficient utilization,data management,and breaks through the data sharing bottleneck in the university,improves the utilization value of university big data,and provide service to the vast number of teachers and students in more intelligent and efficient way.