Testing And Evaluating System Reactions On The Basis Of Injected Misconfigurations

As the software systems become more and more complex,so does their configura-tions.Nowadays,misconfiguration has become one of the major causes of failures.But most of state-of-art researches concentrate on automatically diagnosing the misconfigu-rations,and there is seldom work on system reactions to misconfigurations.Targeted at the drawbacks of these researches,we conduct a empirical study on configurations from widely-used open source software systems,and we proposed a more comprehensive type classification for configuration options.Based on this classification,we design and im-plement our tool ConfVD,to infer the configuration constraints and generate misconfig-urations.ConfVD injects these misconfigurations into software systems and analyze the system abilities according to their reactions to misconfigurations.We classify our work into the following aspects:1.We conduct an empirical study on 8 widely-used open source software systems,including Squid Nginx,Redis,Nagios,Lighttpd(core),Puppet,SeaFile and Vsftpd.We propose a configuration type classification which can well represent 96.5%of configu-ration options in realistic.Based on the classification,ConfVD can infer syntactic,se-mantic and format-related constraints for configuration.After verification,our inferred constraints are consistent with 91%of options in our study.2.On the basis of the constraints ConfVD infer,ConfVD can generate syntactic,semantic and format-related misconfigurations.Then,ConfVD uses the abstract represen-tation for configurations to automatically inject misconfigurations into different software systems.ConfVD uses ABNF to describe the constraints within configuration options,so that fine-grained constraints can guide ConfVD to generate more comprehensive miscon-figurations and get more various injection reaction results.3.We design and implement a testing framework in ConfVD aiming at testing the system reactions to misconfigurations.According to the results from the test,we classify system reactions into 6 main types to leverage the ability in diagnosing misconfigurations.Based on natural language processing techniques,classification can be done automatically by ConfTest.After analyzing the results,we found that 15%of system reactions give rise to the difficulties in diagnosing misconfigurations.Among these misconfigurations,path-related ones are most difficult to deal with.Our experiment data reveals that the configuration correctness check in software startup could effectively reduce the risk of bad system reactions.Through the comparison with the formers,ConfVD can find 3 times bad reactions more than ConfErr,because of the benefit from constraints we found in configurations.For coarse-grained constraints used in SPEX,our misconfigurations guided by more fine-grained constraints can find 18.6%more bad reactions,which makes comprehensively evaluating system reaction ability possible.