Research On Android Automation Test Based On Convolutional Neural Network

In traditional Android automation test,there are mainly three kinds of methods to achieve the purpose of control and testing,including use uiautomator to capture the user interface layout of Android operating system to achieve precise control,the second is to use the Android built-in application instance to realize direct control,the last one is to use monkey command to realize performance testing with random operations.But as the testing requirements change from time to time and software quality standards improve continuously,these approaches are somewhat compromised.In the first method most properties of targeting widget will change once the system got upgraded,which will increase the maintenance effort as consequence,what's more the uiautomator cannot work properly when there is a dynamic activity in user interface.The second method cannot cover the simulated user operation experience.The third one is random operation and unable to achieve accurate control,which lead to low test efficiency.Considering most scenarios mentioned above that we need to click a particular area in interaction interface,so we try to integrate CNN(Convolutional Neural Network)based object detection techonology into current automation test framework to replace the traditional operation method.On one hand,the detection model trained by many samples not only could correctly identify the target area and achieve the purpose of accurate control,but also reduce the burden of regular maintenance work with its generalization ability,and could cover the user operation experience at the same time.On the other hand,we can have 100% effective operations in performance testing to improve test efficiency.In recent years,although convolutional neural network has achieved remarkable achievements in image classification and recognition,from R-CNN,Fast R-CNN,Faster R-CNN,to more and more rapid and accurate target detection methods like YOLO(You Only Look Once)and SSD(Single Shot Multi Box Detector),there is no relevant public technical solution in the field of Android automation testing.For model selection,CNN based object detection technology mainly has two kind of methods,one is based on the region proposal and feature extraction method,the other is based on the regression method,considering the efficiency and accuracy of automation test this study compared two kinds of typical models Faster R-CNN and SSD,and selected the former one after a series of experiments,however we still see there has possibility to improve detecting small object in scenario of Android automation testing,so we propose an improved model which can enhance the detection capability for small object by modifying the model feature extractor,and this has been proved to be valid after few experments;For the platform,we chose the GPU version Tensorflow,which is a popular deep learning platform at present,and could provide rich and mature upper and lower application interfaces for programming;In terms of training,by studying the steps of test cases we capture all the target images as raw samples,and label them into different categories based on the test case logic and the characteristics of the target object,the detection model can identify the right target and classification with hundreds of thousands of times training;In terms of application,after model is integrated into existed test framework,test execution speed is slightly down but still acceptable,execution is much more stable(won't be affected by dynamic activity),the accuracy is slightly lower(this situation will be improved by collecting more and more samples and keep training the model,),the effectiveness of the performance test has been improved a lot(the efficiency can achieve 100% comparing with the original completely random operations),on the other hand,we no longer need extra resource to do complicated maintenance work,all we need is to collect more samples,label the right classifications,and keep training the model.From this application research we can find that the CNN based object detection technology has improved a lot on the efficiency of Android performance automation test comparing with traditional Android automation test,although it is slightly inferior in the Android functional automation test at current stage,but this issue can be fixed through continuous optimization of detection model and it would has stronger ability to handle this kind of object detection problem.Therefore,the CNN based object detection techonology has a good application prospect in Anroid automation testing.