The Design, Implementation And Application Of A Distributed Data Collection And Monitoring System

During the course of research on computer architecture and performance, the datacollection and monitoring system plays a very important role all the time. It can obtain datafrom software and hardware while the system is running. Such data may reflect theperformance, workload and efficiency of systems, and give some helpful guidance forimproving existing systems or designing new architectures. At present, the study on centralizedpersonal computing and storage paradigm with the main body of data and computing centerhas become a hot spot in architecture researches. In this paradigm, computing and storageresources are centralized on the server. Users can have their own safe personal computingenvironment by using thin clients connected to the server and virtualization technologies. Thecentralized paradigm aims to improve resource utilization and reduce costs by enablinglarge-scale resource sharing, which challenges the design of the server. However, the resourcesand workloads of such server are actually the aggregation of many existing personal resourcesand workloads. Therefore, the studies on characteristics of resources usage, applications anduser activities of existing personal computers can give some quantitative guidance for thedesign of the server in centralized paradigm. Thus we must collect and monitor the data fromlots of existing networked personal computers, and during this process the types of data andusers monitored may vary. For these needs, this paper designs and implements a dynamicupdatable distributed data collection and monitoring system, and analyzes the collected data indetail.The main contributions of this paper are listed below:1. We study the architectures and techniques of existing distributed monitoring systemsand discuss key issues about the design of the system for collecting and monitoringthe data from personal computers in network environment. Basing on the analysis ofexisting systems, we present the architecture of a dynamic updatable distributed datacollection and monitoring system DDCS. This architecture, on the basis ofManager/Agent model, adopts the independence of data collection modules andserviced monitoring process. It meets the needs of collecting the variety of the types ofdata during long-term researches.2. On the basis of the dynamic updatable architecture, we implement the distributed datacollection and monitoring system: DDCS. Through dynamic update protocol, DDCSimplements dynamic updatable mechanisms of parameters, collectors and system.With many technologies, DDCS achieves low overhead. In addition, DDCS supportsmultilayer and across-domain data collection. DDCS provides collector developmentkit for system users.3. By installing DDCS on 40 personal office computers, we obtain lots of data. Afteranalyzing these data from three aspects: resource usage, office applications and useractivities, we obtain many results. We find that the CPU average utilization of officecomputer is only 12.9%, the average unused RAM is over 30%, and the averagebandwidth utilization of disk and network I/O is under 1%.The DDCS system, the collected data and the analyzed results presented in this paper willbe used in researches on the strategies of organizing and assigning resources and thescheduling algorithms for tasks of the server in centralized personal computing and storageparadigm.