The Design And Implementation Of Microsoft IScope System KPI Metrics Module

With the rapid development of the Internet, the data grows explosively. More and more companies rely on the results of such massive data analysis for their business decisions. A lot of massive data analysis systems have been developed in these years, such as RDBMS, Hadoop system of Apache and Hive system of Facebook. But with the increasing of data size and the improvement of real-time demand, these data analysis systems can not include both the two features. The iScope system whose development I involved in for my internship in Microsoft can solve the problem to some extent. In one hand, iScope is based on Cosmos which is the distributed system developed by Microsoft. It has Strong extensibility. In the other hand, iScope skip the job management of MapReduce, and call on the data directly. It decreased time of the jobs running and realized the interactive feature.As a real-time interactive system, iScope needs to be measured in KPI to judge the performance.of it. The KPI metrics can help users know more about the system and help the developers to make the necessary improvements. The KPI metrics module which I am responsible for in my internship use the mothod of key events record and the log base analysis to solve the problem of the iScope system KPI metrics. This paper introduces the iScope system KPI metrics module in detail. The iScope system KPI metrics module includes two parts. Firstly, we print the logs of the key events in iScope system, and upload these logs to the Cosmos system. Secondly, we do some computation and statistics jobs for the key events logs, and make the jobs running in the Scope system regularly. These jobs produce some data files which record the analysis data for users to view and generate reports.