Design And Implementation Of Distributed Systems Tracing Infrastructure Based On Adaptive Sampling

Contemporary Internet applications are usually complex and large-scale distributed systems and the runtime environments of them are complicated. When there is a performance problem, it is difficult to troubleshoot the root cause of the problem. The existing distributed tracing systems solves this problem, but all of them adopt the fixed sampling probability, can not adjust the sampling rate according to application’s busy degree. On the other hand, the typical design scheme of distributed tracing system does not support complex sampling. So this paper presents a design based on sampling and an adaptive sampling algorithm, and then implements a distributed tracing system based on adaptive sampling, which is used to help operation engineer understand the distributed system’s behavior and analyze the root cause of performance problem.Firstly, this paper analyzes the related technologies of typical distributed tracing systems. Then we design a distributed tracing solution based on sampling and an adaptive sampling algorithm, which is based on the functional requirements and performance requirements of contemporary Internet applications. Finally, we implement the tracing systems according to the design. The system is of low overhead, application-level transparency and scalability. And we proposed the effectiveness of the tracing system based on adaptive sampling by some experiments.The proposed tracing design based on adaptive sampling makes the sampling volume controllable, and help to analyze the performance problem with low overhead. Finally, the system performed well in the real industrial environment.