Research And Implementation Of Field-Oriented Data Analysis Platform

With the development of information technology, it has been widely used in our daily life, so it gets more and more focus and investment by enterprises. Simultaneously with the development of the enterprise business, the amount of data within the enterprise is growing. Therefore, how to effectively analyze these data, dig out valuable information, provide a strong basis for business decisions,which can directly relate to the ability of the business to survive in the market. The traditional data mining platform, because of its accuracy, is widely used in business data analysis, such as the market price forecasting, employee turnover behavior analysis.However, with the explosive growth of data, the traditional data mining platform has been unable to process the data in a short time. On the other hand, with the internal business diversification, the cost of integrating new components in traditional data mining platform increases.In order to solve the above problems, this paper proposed and implemented a new data analysis platform for field-oriented architecture. The architecture is based on the mass storage HDFS, MapReduce framework, making massive data processing become feasible. And by introducing a scalable workflow engine with a plug-in framework, making the scalability greatly enhanced. In summary, the main content of the paper includes:1. The architecture design of platform:Based on the needs of the oil sector companies, complete the architecture design of the platform, following detailed design of each module.2. Unified metadata specification:Define a unified metadata specification, which each module of platform use to communicate between each other.3. Dynamicly deployed plugin framework:The use of OSGI(Open Service Gateway Initiative) specification gives platform the ability to integrate other capabilities into the platform. Besides, an integrated mechanism of workflow engine gives the platform the ability to organize effectively, enable the platform can support complex business processes.4. Finally, we apply the platform into a scenario, which shows up the high efficiency, high scalabilty, and supporting of complicated process,the feasible of the platform.