Design And Implementation Of A Distributed Open Platform For Big Data Services

As modern system frameworks become larger,big data platform need to handle different types and sizes of data.Both a traditional relational database is required to store structured information,and a non-relational database is required to store log files for mass data analysis.This requires application systems which developed based on big data platforms to have different types of connections,which means the application itself is responsible for the connection and processing logic of multiple data sources.There will be the following problems:1)Developers need to manage multiple databases of different architectures on the big data platform,understand the underlying specific details and functional features,and maintain independently.The respective data connections and access interfaces of multiple data sources cannot achieve unified management of multiple data sources;2)Massive data is stored decentralized in the big data platform.Developers not only need to understand the specific details of the underlying databases,functional features,also need to know the storage location of the data to get the data products you need and build your own personalized service.3)A variety of data sources with different architectures under the big data platform maintain their respective rights management information.Developers need to understand this information and apply relevant permissions can query the data,and data owners need to repeatedly confirm these permission applications to prevent leakage of sensitive data.Based on the above problems,this subject intends to build a standardized open platform for big data service capabilities,which will open the data stored in various underlying data sources to the outside world.It can flexibly and quickly adapt to various application scenarios,unify data export,and enhance data input.While providing a unified permission managcment subsystem,allowing application developers to focus on the development of system business logic based on data services to achieve rapid iteration of business.First of all,the system needs analysis is based on the problem.The platform needs to have multiple data source support,provide unified data access interfaces,data service openness,data service management,and unified user rights management.Non-functional requirements such as stability.Then based on the analysis of system requirements,the overall design of the system is completed.The main research contents include the following three parts:1)Dynamic service open subsystem based on multiple data sources to achieve a unified database of multiple different architectures under the big data platform.Hide its underlying specific details and functional characteristics,make it externally appear as a single database,and provide a unified ANSI SQL data access interface to the outside world,and provide a data service based on a unified access interface to open business processes and convert ANSI SQL into RESTful services,which define and publish data services,and open data as services;2)The service management subsystem based on API gateways implements the full life-cycle management of data services,and builds a standardized service capability open platform to unify data Service entrance to realize the opening of big data products;3)Unified user rights management subsystem to achieve unified role-based rights management.Finally,the implementation of the system is completed based on the system design.This paper designed and finally implemented a distributed big data service open platform,which abstracts multiple data sources under the big data platform into a data source that provides a unified data access interface,and is based on the unified data access interface(ANSI SQL)Constructing a standardized service open platform and opening data as a service to the outside world,which has greatly reduced the extra development cost of applications based on big data platforms to handle multiple data sources,while improving the efficiency of data use.