The Design And Implementation Of Multidimensional Evaluation System For Android Automated Testing Tools

With the rapid development of mobile Internet,Android devices have been widely used.However,due to the open source nature of Android,there are great differences between Android devices,which makes the fragmentation of the Android ecosystem become a growing problem.Therefore,testing is becoming more and more important for Android applications.The fierce competition between applications has led to ever-shortening development cycles.It also makes users demand more and more of the application.Manual testing cannot meet the current development needs due to the high time cost.Therefore,the automatic test tool industry is thriving.Although most tools claim to be more effective,their testing effects do not consider the evaluation metrics,application sets,and equipment comprehensively enough,causing the results are not objective enough.Therefore,this thesis proposes a more comprehensive and objective measurement system for automatic testing tools.This system is conducive to the horizontal comparison of tools,and provides a reference for the development of subsequent automatic testing tools.In this thesis,we investigate the existing automatic testing tools,abstracts five multidimensional evaluation metrics,including exceptions,mutation,code coverage,UI component coverage and fragmentation support.The thesis designs detection methods for the above indicators.The problem of incomplete evaluation metrics is solved.The system has a scalable set of applications to be tested,which includes the original application after instrument and the corresponding mutated applications.The user only needs to provide the application source code to expand the application set.By implementing the corresponding interfaces,the system can execute the tools to be tested on the real cluster.The system monitors the automatic test tools and devices during the test execution,collects the files generated during application runtime,device logs and log information of the tested tool.These data are stored as raw data and as input to the subsequent analysis module.According to the multidimensional evaluation index formula proposed in this thesis,the raw application and mutated application data are analyzed,the specific scores of the tested tools can be obtained.The back-end of the system is completed using the Spring Boot framework.and connect to the device to be tested through ADB.The front-end is implemented using React.The results are displayed graphically through the Echarts framework.Users can understand the test results from the images in a simple and intuitive way.In this thesis,seventeen open source applications are selected for automatic instrument and mutation operation.Except for one application that failed to compile due to system environment factors,all the other applications are successfully executed the operation.Including the mutated applications a total of 1089 applications sets have been formed to be tested.The system has been adapted to five well-known automatic testing tools.In this thesis,we use the system to test all these tools.A total of 5040 test tasks of the automatic test tools were performed on 6 devices.The system can effectively manage the devices and test tools during the testing process.All features including test data collection,log collection,monitoring performed normally.The system was used to analyze the collected data after the tests.The automatic test tool was successfully evaluated using the multidimensional evaluation indexes proposed in this thesis.The evaluation results clearly show the test performance and specific scores of the automatic test tool in five dimensions.