Research And Development Of Key Technology Of Data Bus System Based On Kafka

Today's era is an era of information explosion,every corner of production and life is full of all kinds of information,enterprises and institutions have a natural advantage in information acquisition,is the owner of all kinds of data.Through the efficient sharing of data,the potential value of data can be excavated to the maximum extent,which is conducive to scientific management,innovative management and long-term planning of enterprises and institutions.However,the existing data sharing system has a variety of disadvantages,either the old technology can not adapt to the era of big data data sharing needs,or the price is high and cost-effective,therefore,a support mass data efficient sharing,and low cost of the data bus system is extremely necessary.This paper designs and develops a data bus system based on Kafka based on various open source technologies.On the system requirements analysis,detailed elaborated the user for data acquisition,real-time computing,data dump,bus metadata management,operational monitoring of the functional requirements and performance requirements,functional module chart is given,the system architecture diagram,data flow diagram,in detail elaborated the system design and implementation,the core algorithm and code are given,the system is completed by the methods of black box testing,the control demand design test to verify the function of each module and performance.The system adopts the micro-service architecture.Under the JVM development platform,Flume is used as the data acquisition component,Kafka as the data pipeline,kettle as the ETL tool,and Spring Cloud as the implementation framework for the micro-service development.JAVA code is used to realize various functional modules.