The Design And Implementation Of Data Property Management System

Web applications in their early days were designed to pursuit higher storage density over a shorter period of operation time.Today things are a little different.The Internet is now featuring bigdata.With the growing number of fields that the Internet has an impact on in everyday life,an increasing amount of information is being recorded for further mining and analysis.In recent years,Ximalaya successfully started quite a few business festivals like 123 festival and 66 festival.Millions of attendances are involved.Every user operation is written into some data warehouse.Mining,calculating and analyzing audiences' likes and dislikes would be very profitable.The choice of data warehouses(DWs)must satisfy at least one of the following rules:high compression rate and/or low response time.Different business scenarios have different space and time requirements,which leads to different DW choices.With the bloom of business scenarios,a growing number and species of DWs have been introduced for storage purposes.What follows is that analyzers must switch between a lot of DWs for a comprehensive understanding of data from different DWs.This system deals with the inconvenience caused by accessing and switching between different DW-reading systems.Users are able to browse different DWs within this system.Specifically,a user can put 2 tables near each other,even if one of them comes from MySQL,and the other one comes from Hive.This thesis introduces different bigdata frameworks and their corresponding writing technologies like HDFS,HBase,Hive,MySQL,ES,Kafka.This thesis explains how a data property management system is implemented based on the introduction of those technologies.The system has 3 layers:the presentation layer,the business logic layer,and the data-accessing layer.The presentation layer uses a frontend framework called AntD.The business logic layer uses Spring Boot.The data-accessing layer uses different open-source Java libraries because we need to fetch data from different kinds of DWs.The data property management system is now running online.With the evolution and development of business scenarios,there might be more requirements.