Design And Implementation Of Resource Monitoring System In Distributed Systems

With the development of technology,more and more data are being generated.At the same time,the request for dealing with such a big amount of data becomes more urgent.The general solution towards this problem is to use clusters which are composed with commodity computers.As commodity computers are less stable than server grade hardware,monitoring becomes especially important.Besides,monitoring is beneficial for resource arrangement as well as performance analysis.Based on existing data transfer protocols,an improved data transfer protocol has been proposed.It can achieve better balance between data fidelity and communication overhead.The resource monitoring system consists of monitoring module,transfer module and consumer module.Monitoring module runs on the node of the distributed system generating the resource monitoring data.The delivery of resource monitoring data is controled by the proposed data transfer protocol.Kafka cluster acts as the tranfer module in the monitoring system.By doing so,it can improve the reliablity and throughput of data transfering.To provide easier way of using the monitoring resource data,consumer module consumes the data in Kafka cluster and writes it to local database.Experiments show that the proposed data transfer protocol achieves better balance between data fidelity and communication overhead.The resource monitoring system is scalable and monitors the distributed system efficiently.It helps the system administrator take better control of the system.