Research And Application Of BESIII Distributed Computing Accounting System Based On MongoDB

Experimenter in Beijing Electron Spectrum III(BESIII) needs to process and analyzethe PB massive data every year. Thus, in order to meet the challenges posed by the massivedata to the existing computing and storage resources, BESIII, working with severalcooperation organizations, has constructed the unified distributed computing platformsince2011. Further, in order to improve its using efficiency, a massive grid system requiresan effective record and analysis for the usage of resources. The Grid Accounting System isjust the system which could record resource using condition under a grid environment. Thesystem not only can record the consumption of resources, but also could put the record intoan effective analysis and use. This paper is mainly about the design and realization of theaccounting system based on the BESIII distributed computing platform. The contents areas follows:I. To study the function module design of the accounting system. When design theinformation gathering module, the author proposes a real-time accounting strategy to solvethe latency problem in the traditional log parser strategy, which achieves a quickaccounting. When design the memory module, the author creates an anti-normalizationdata modeling based on MongoDB and conducts data sharding using a compound shardkey, which improves the scalability of the system. At the same time, the replica set alsoenhances availability of the system.I. To study the real-time response to statistical query request. A data compressionalgorithm based on the dynamic time window is proposed after analyzing users’ queryrequests. This algorithm allows compression preprocessing to data according to variousexisting time. The method greatly condenses the data size and reduces the computingvolume, which lifts efficiency in statistical query. III. To realize the function module. The author builds MongoDB cluster as the bottomstorage platform, and uses matplotlib to enrich the query results. Besides, runit employinghas realized the persistence service and improved the robustness of the system. Finally, todo a comparision test of the system performance.