Design And Implementation Of Data Acquisition System On Big Data Platform

With the support of China's policy,large data technology has been widely used in the government and enterprise projects,but as the fields involved in the large data are more and more extensive,the processing business is becoming more and more in-depth,and the data types to be collected and processed are becoming more and more complex and diverse.In recent years,the data types of the government and enterprise customers have changed from the traditional relational data to the non relational data,and have the characteristics of high speed,large capacity,diversity and high value.This brings new challenges to the large data collection,but the data collection can not meet the new collection needs at the present stage,and the data collection is also faced with various data collection.For example,the data collection is relatively simple,the acquisition process is more complex,can not provide a friendly user experience,can not make a good package of data collection process,the monitoring of the collection process is relatively lack,and can not provide users with more detailed information on the acquisition of state information and other problems.In this paper,a data acquisition system will be designed and developed.This collection system will be a subsystem of the large data platform of the telecom sky wing Feilong.It can solve the problems related to data acquisition in the current industry.This collection system can be structured data,semi-structured data and non knot.The user can also make a single key deployment of the collected processing components,and can configure the collection components conveniently.This system also implements the function of visual collection process for users,and provides convenient monitoring and management.It provides a new solution for large data collection.At present,the project has completed the research on the development of large data collection at home and abroad,and the analysis of the advantages and disadvantages of the major data collection products.The functional requirements and non functional requirements of the collection system are completed,and the power point and non functional requirements of the acquisition system are clarified.Secondly,it is completed.Based on the overall design and detailed design of this collection system,the system architecture is defined and the system is divided into detailed modules.The custom development of flume and the design and development of user interaction module are completed,so that users can easily complete the deployment of the components without contact with the underlying operation of the Linux.The configuration task of each collection module completes the design and development of the visual collection workflow module,and provides users with a perfect monitoring function.The main contents of my work are as follows:the research work on large data collection at home and abroad is completed,the requirement of the collection system is clarified,and the overall design and detailed design of this system are carried out,and the custom flume module,the deployment processing module module,the configuration collection module and the visual collection workflow module are completed.And the code development task of the monitoring module;finally,the detailed test of the acquisition system is carried out,and the possible problems in the test are analyzed,and the corresponding solutions are given.The data acquisition system has completed all the design and development work,and has passed the detailed test link.As a subsystem of the telecom Tianyi flying dragon data platform,this collection system has been successfully deployed in the production environment of many government and enterprise customers,and bears the data collection task of TB/PB level.The system can satisfy the requirements of data collection for government and enterprise customers.