An Automatic I/O Congestion Control Mechanism Based On Deep Q-learning

Thera are a large number of applications' I/O requests to contend for limited resources which cause I/O congestion similar to network in parallel distributed file system when it reaches a large scale,and lead to decreased throughput and uncontrollable delay.This paper researches I/O congestion control in Lustre distributed file system.Existing Lustre does not take much consideration into I/O congestion problem caused by large-scale I/O operations and lacks necessary regulation thus leading to low efficiency.Manual I/O congestion control on such a large scale cluster storage system is extremely unrealistic,so a solution for automatic I/O congestion control for global QoS is essential and significant especially when it's extended to Exa-scale.This thesis proposes an automatic I/O congestion control solution AIOCC.AIOCC designs a server-side I/O request scheduler to maximize the I/O efficiency,which implemented by the token bucket policy,and broadcast the assigned value to the client correspondingly.The client-side congestion window mechanism adjusts both the number and the rate of sending I/O request according to system status and allocate congestion window on application granularity.In addition,AIOCC uses Deep Q-Learning algorithm to optimize these control parameters for both client-side and server-side automatically.Thus,AIOCC is able to mitigate the I/O congestion,and efficiently improve the system throughput and I/O completion latency in Lustre.Experiments show that AIOCC is able to improve the throughput of some Lustre workload by 34.82%,and I/O completion latency 26.17% at most which proved the ability of AIOCC of alleviating the I/O congestion in Lustre.