Design And Implementation Of A Communication Server To Support Collection Of Multi-source And Heterogeneous Streaming Data

As the widespread application of internet of things technology, it promoted lots of new applications and brought changes to the traditional industry, such as smart home, intelligent transportation, smart cities. Behavioral activities of different industry has been transformed into a dynamic streaming data with new features of real-time, concurrent and heterogeneous, the traditional processing methods and models have been unable to meet access the dynamic streaming data. The main difficulty is that diverse data types and a wide range of data sources:for example, GPS data, vehicle traffic data, and other data collected by sensors and other types of sensory data; data sources include a large number of different sensors and conversion data of variety behavior from transportation, agriculture, health care, e-commerce and other fields.the above data access technology needs lead to two aspects:How to make dynamic heterogeneous streaming data access the center of the cloud including how to transfer a large number of front-end streaming data and how to access different types of dynamic streaming data; How to meet the streaming data communication server for parsing and efficient distribution of data, in order to response large amounts of data access and support to other business in communication server, it requirement efficient data analysis and distribution of data.In the background of the current lack of effective streaming data collection systems, this paper design and implementation of a communication server to support collection of multi-source and heterogeneous streaming data after grasping the network communication protocol, heterogeneous streaming data collection technology and asynchronous IO technology, we call it streaming data communications server. In cloud environment It can support massive data access, parsing and distribution. What’s more,it provides the web application for users to monitor data, and it can help to save data for other applications to use to the streaming data. The main contents include:(1) it designed a data communication protocol to support heterogeneous streaming data collection, including the data frame format to transmit and response, and some reliability and enhanced design. The different sources, different types of data packaged in accordance with the format of the data communication protocol, not only support data communications but support access heterogeneous streaming data.(2) Using asynchronous 10 ways to support high concurrent for streaming data communication server, this paper uses Libevent and Multithreading Connection Pooling and other technology to achieve high concurrency.It can be highly concurrent data parsing and distribution to ensure the system performance after accessing large amounts of data,and it can send the treated message packet to the ActiveMQ server for message transfer.(3) Develop a set of internet of things sensor data delegation service data hosting application prototype system,it is designed on the basic of streaming data communication server of this article has been implemented.It enables users to manage devices and data, it provides a new service support model for different types of sensory data collection below to different users in cloud environment. In addition, through the experimental test, the system can achieve reception capacity of 3000 data per second in general server configuration under 3,000 concurrent connections (3000 terminal equipment) in long-connection.